htcondenser.common module

Functions/classes that are commonly used.

class htcondenser.common.FileMirror(original, hdfs, worker)

Bases: object

Simple class to store location of mirrored files: the original, the copy of HDFS, and the copy on the worker node.

htcondenser.common.check_certificate()

Check the user’s grid certificate is valid, and > 1 hour time left.

Raises:RuntimeError – If certificate not valid. If certificate valid but has < 1 hour remaining.

htcondenser.common.check_dir_create(directory)

Check to see if directory exists, if not create it.

Parameters:directory (str) – Name of directory to check and create. Raises:IOError – If ‘directory’ already exists but is a file.

htcondenser.common.check_good_filename(filename)

Checks the filename isn’t rubbish e.g. blank, a period

Raises:OSError – If bad filename

htcondenser.common.cp_hdfs(src, dest, force=True)

Copy file between src and destination, allowing for one or both to be on HDFS.

Uses the hadoop commands if possible to ensure safe transfer.

Parameters:

• src (str) – Source filepath. For files on HDFS, use the full filepath, /hdfs/...
• dest (str) – Destination filepath. For files on HDFS, use the full filepath, /hdfs/...
• force (bool, optional) – If True, will overwrite destination file if it already exists.

htcondenser.common.date_now(fmt='%d %B %Y')

Get current date as a string.

Parameters:fmt (str, optional) – Format string for time. Default is %d %B %Y. See strftime docs. Returns:Current date. Return type:str

htcondenser.common.date_time_now(fmt='%H:%M:%S %d %B %Y')

Get current date and time as a string.

Parameters:fmt (str, optional) – Format string for time. Default is %H:%M:%S %d %B %Y. See strftime docs. Returns:Current date and time. Return type:str

htcondenser.common.time_now(fmt='%H:%M:%S')

Get current time as a string.

Parameters:fmt (str, optional) – Format string for time. Default is %H:%M:%S. See strftime docs. Returns:Current time. Return type:str