htcondenser.jobset module

Class to describe groups of jobs sharing common settings, that becomes one condor submit file.

class htcondenser.jobset.JobSet(exe, copy_exe=True, setup_script=None, filename='jobs.condor', out_dir='logs', out_file='$(cluster).$(process).out', err_dir='logs', err_file='$(cluster).$(process).err', log_dir='logs', log_file='$(cluster).$(process).log', cpus=1, memory='100MB', disk='100MB', certificate=False, transfer_hdfs_input=True, share_exe_setup=True, common_input_files=None, hdfs_store=None, dag_mode=False, other_args=None)[source]

Bases: object

Manages a set of Jobs, all sharing a common submission file, log locations, resource request, and setup procedure.

Parameters:
  • exe (str) – Name of executable for this set of jobs. Note that path must be specified, e.g. ‘./myexe’
  • copy_exe (bool, optional) – If True, copies the executable to HDFS. Set False for builtins e.g. awk
  • setup_script (str, optional) – Shell script to execute on worker node to setup necessary programs, libs, etc.
  • filename (str, optional) – Filename for HTCondor job description file.
  • out_dir (str, optional) – Directory for STDOUT output. Will be automatically created if it does not already exist. Raises an OSError if already exists but is not a directory.
  • out_file (str, optional) – Filename for STDOUT output.
  • err_dir (str, optional) – Directory for STDERR output. Will be automatically created if it does not already exist. Raises an OSError if already exists but is not a directory.
  • err_file (str, optional) – Filename for STDERR output.
  • log_dir (str, optional) – Directory for log output. Will be automatically created if it does not already exist. Raises an OSError if already exists but is not a directory.
  • log_file (str, optional) – Filename for log output.
  • cpus (int, optional) – Number of CPU cores for each job.
  • memory (str, optional) – RAM to request for each job.
  • disk (str, optional) – Disk space to request for each job.
  • certificate (bool, optional) – Whether the JobSet requires the user’s grid certificate.
  • transfer_hdfs_input (bool, optional) – If True, transfers input files on HDFS to worker node first. Auto-updates program arguments to take this into account. Otherwise files are read directly from HDFS. Note that this does not affect input files not on HDFS - they will be transferred across regardlass.
  • share_exe_setup (bool, optional) – If True, then all jobs will use the same exe and setup files on HDFS. If False, each job will have their own copy of the exe and setup script in their individual job folder.
  • common_input_files (list[str], optional) – List of common input files for each job. Unlike Job input files, there will only be 1 copy of this input file made on HDFS. Not sure if this will break anything...
  • hdfs_store (str, optional) – If any local files (on /user) needs to be transferred to the job, it must first be stored on /hdfs. This argument specifies the directory where those files are stored. Each job will have its own copy of all input files, in a subdirectory with the Job name. If this directory does not exist, it will be created.
  • other_args (dict, optional) – Dictionary of other job options to write to HTCondor submit file. These will be added in before any arguments or jobs.
Raises:
  • OSError – If any of out_file, err_file, or log_file, are blank or ‘.’.
  • OSError – If any of out_dir, err_dir, log_dir, hdfs_store cannot be created.
add_job(job)[source]

Add a Job to the collection of jobs managed by this JobSet.

Parameters:

job (Job) – Job object to be added.

Raises:
  • TypeError – If job argument isn’t of type Job (or derived type).
  • KeyError – If a job with that name is already governed by this JobSet object.
generate_file_contents(template, dag_mode=False)[source]

Create a job file contents from a template, replacing necessary fields and adding in all jobs with necessary arguments.

Can either be used for normal jobs, in which case all jobs added, or for use in a DAG, where a placeholder for any job(s) is used.

Parameters:
  • template (str) – Job template as a single string, including tokens to be replaced.
  • dag_mode (bool, optional) – If True, then submit file will only contain placeholder for job args. This is so it can be used in a DAG. Otherwise, the submit file will specify each Job attached to this JobSet.
Returns:

Completed job template.

Return type:

str

Raises:

IndexError – If the JobSet has no Jobs attached.

setup_common_input_file_mirrors(hdfs_mirror_dir)[source]

Attach a mirror HDFS location for each non-HDFS input file. Also attaches a location for the worker node, incase the user wishes to copy the input file from HDFS to worker node first before processing.

Parameters:hdfs_mirror_dir (str) – Location of directory to store mirrored copies.
submit(force=False)[source]

Write HTCondor job file, copy necessary files to HDFS, and submit. Also prints out info for user.

Parameters:force (bool, optional) – Force condor_submit
Raises:CalledProcessError – If condor_submit returns non-zero exit code.
transfer_to_hdfs()[source]

Copy any necessary input files to HDFS.

This transfers both common exe/setup (if self.share_exe_setup == True), and the individual files required by each Job.

write(dag_mode)[source]

Write jobs to HTCondor job file.