Multi Dags

Note: not yet complete!

Because there's no requirement that all nodes in a given DAG file be connected, it's pretty simple to allow multiple DAG files to be run by a single DAGMan invocation. The main issue is making sure that you don't have name collisions between nodes in different DAG files (if there are name collisions within a DAG file, that's a user error...).

To avoid the name collision problem, we simply rename the nodes (add details here).

Once parsing has happened, we no longer have any record of which DAG file a given node came from -- essentially we have a DAG structure that could be created by a single DAG file with appropriate node names. One consequence of this is that, if a rescue DAG is generated, we get a single rescue DAG file that incorporates information for the entire structure corresponding to all of the input files.

Things to mention:

  • node name munging
  • you get a single rescue DAG
  • primary DAG file