htcondenser.common module¶

Functions/classes that are commonly used.

class htcondenser.common.FileMirror(original, hdfs, worker)[source]¶

Bases: object

Simple class to store location of mirrored files: the original, the copy of HDFS, and the copy on the worker node.

htcondenser.common.check_certificate()[source]¶

Check the user’s grid certificate is valid, and > 1 hour time left.

Raises:	`RuntimeError` – If certificate not valid. If certificate valid but has < 1 hour remaining.

htcondenser.common.check_dir_create(directory)[source]¶

Check to see if directory exists, if not create it.

Parameters:	directory (str) – Name of directory to check and create.
Raises:	`IOError` – If ‘directory’ already exists but is a file.

htcondenser.common.cp_hdfs(src, dest, force=True)[source]¶

Copy file between src and destination, allowing for one or both to be on HDFS.

Uses the hadoop commands if possible to ensure safe transfer.

Parameters:	src (str) – Source filepath. For files on HDFS, use the full filepath, /hdfs/... dest (str) – Destination filepath. For files on HDFS, use the full filepath, /hdfs/... force (bool, optional) – If True, will overwrite destination file if it already exists.

htcondenser.common.date_now(fmt='%d %B %Y')[source]¶

Get current date as a string.

Parameters:	fmt (str, optional) – Format string for time. Default is %d %B %Y. See strftime docs.
Returns:	Current date.
Return type:	str

htcondenser.common.date_time_now(fmt='%H:%M:%S %d %B %Y')[source]¶

Get current date and time as a string.

Parameters:	fmt (str, optional) – Format string for time. Default is %H:%M:%S %d %B %Y. See strftime docs.
Returns:	Current date and time.
Return type:	str

htcondenser.common.time_now(fmt='%H:%M:%S')[source]¶

Get current time as a string.

Parameters:	fmt (str, optional) – Format string for time. Default is %H:%M:%S. See strftime docs.
Returns:	Current time.
Return type:	str