Dag Jobs On Same Machine

A question that is often asked about DAGs is this: how can I make two or more DAG node jobs run on the same machine? (Typically this is because the first job generates large temporary files that are to be used by the second job, and it is desirable to avoid having to transfer those files between machines.)

This document presents a way to ensure that consecutive DAG nodes run on the same machine, without pre-selecting a specific machine.

(Note: this document refers to a jobs for a parent and a child node running on the same machine; but the scheme can be easily extended to any number of descendants.)

The basic scheme :

The parent node has a POST script that determines which machine the node job ran on; this script then outputs a file containing the machine name, and that file is incorporated into the submit file of the child node job (and any subsequent jobs that should run on that machine) in such a way that the child job is required to run on that machine.

This scheme takes advantage of claim leases (see http://research.cs.wisc.edu/htcondor/manual/v8.5/3_5Policy_Configuration.html#35520 ), which DAGMan automatically sets to last for 20 seconds after the end of a node job, if the node has children. The claim lease allows a second job from the same user to run on the execute machine without having to go through a negotiation cycle.

(Note that the duration of the claim lease can be changed by changing the setting of the DAGMAN_HOLD_CLAIM_TIME configuration macro, as in the example below.)

Caveats :

  • It is always possible for another job submitted by the same user to "steal" the machine before the child job runs on it. (In other words, the child job is required to run on the designated machine, but that machine is not required to run only the child job.) (The job that does the "stealing" could be an entirely separate job submitted by the same user, or another job from the same DAG, if that job is not assigned to a specific machine.)

  • (If DAGMAN_HOLD_CLAIM_TIME is sufficiently long, it is not possible for a job from a different user to "steal" the machine.)

  • For this scheme to work, the child job must have requirements that will match whatever machine the parent job runs on. (Ideally, the two jobs should have identical requirements.) If the child job's requirements do not match the machine the parent ran on (for example, the child job requires more memory), the child job will never run, and, therefore, the DAG will never complete.

  • Also, if the relevant machine encounters some problem that prevents the child job from running, the DAG will never complete, because the child job is not allowed to be run by any other machine.

  • The jobs to be run on the same machine should belong to nodes that have an immediate parent/child relationship in the DAG -- otherwise the claim lease is likely to expire before the child job can run, if the machine is not "stolen" by another job.

  • The submit file include feature that this example uses was added in version 8.3.5. If you are running an earlier version of HTCondor, you can use a similar scheme, but the parent node POST script would have to write out the entire submit file for the child node job.

Example :

(This is a shortened version of the example in the attached tar file.)

# File: example.dag

  config example.config

  job A0 nodeA.sub
  vars A0 node="$(JOB)"
  script post A0 post.pl B0 $JOBID

  job B0 nodeB0.sub
  vars B0 node="$(JOB)"

  parent A0 child B0

# File: example.config

  DAGMAN_HOLD_CLAIM_TIME = 60

# File: nodeA.sub

  executable = /bin/hostname
  output = $(node).out
  queue

# File: nodeB0.sub

  executable = /bin/hostname
  output = $(node).out
  # Unfortunately, we can't use $(node) in the include file name.
  include : B0.inc
  requirements = TARGET.Name == $(my_machine)
  queue

# File: post.pl

  #!/usr/bin/env perl

  $outfile = $ARGV[0] . ".inc";

  if (-e $outfile) {
         system("rm -f $outfile");
  }

  open(OUT, ">$outfile") or die "Couldn't open output file $outfile: $!";
  $host = `condor_history $ARGV[1] -af LastRemoteHost -limit 1`;
  chomp $host;
  print OUT "my_machine = \"$host\"\n";
  close(OUT);

On *nix systems, you should be able to copy and paste the example above, or download the attached example tar file, and run the example DAG without modification. On Windows you will have to make some small changes to the example for it to work.

Attachments:

  • same_machine_example.tgz 938 bytes added by wenger on 2016-Aug-11 22:34:09 UTC.
    Example DAG that forces each child node to run on the same machine that its parent ran on.