Python Condor Wisdom

Using the python-condor library

python-condor is a wrapper module containing mechanisms for interacting with the HTCondor system. It is included by default in 7.9.4.

Basics - Job Submission

  1. Begin by importing the condor and classad modules from a python interpreter.
    Note: some distributions require you import the module as htcondor

  2. Declare a collector with an IP address/hostname.
  3. Locate a scheduler by applying the 'locate' method on the collector object.
  4. Parse a classad file with the classad library's .parse() method. (Note: these are raw classad files. They are similar to submit files, but vary in interesting ways. condor_q -l xx.xx will display the classad data for a particular job.)
  5. Using the .submit method to submit the file to the appropriate scheduler.
    [bbockelm@example python-condor]$ python
    Python 2.6.6 (r266:84292, Jun 18 2012, 09:57:52)
    [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import condor
    >>> import classad
    >>> coll = condor.Collector("red-condor.unl.edu")
    >>> results = coll.query(condor.AdTypes.Startd, "true", ["Name"])
    >>> len(results)
    3812
    >>> results[0]
    [ Name = "slot1@red-d20n35"; MyType = "Machine"; TargetType = "Job" ]
    >>> scheddAd = coll.locate(condor.DaemonTypes.Schedd, "red-gw1.unl.edu")
    >>> scheddAd["ScheddIpAddr"]
    '<129.93.239.132:53020>'
    >>> schedd = condor.Schedd(scheddAd)
    >>> results = schedd.query('Owner =?= "cmsprod088"', ["ClusterId", "ProcId"])
    >>> len(results)
    63
    >>> results[0]
    [ MyType = "Job"; TargetType = "Machine"; ServerTime = 1356722353; ClusterId = 674143; ProcId = 0 ]
    >>> condor.param["COLLECTOR_HOST"]
    'hcc-briantest.unl.edu'
    >>> schedd = condor.Schedd() # Defaults to the local schedd.
    >>> results = schedd.query()
    >>> results[0]["RequestMemory"]
    ifthenelse(MemoryUsage isnt undefined,MemoryUsage,( ImageSize + 1023 ) / 1024)
    >>> results[0]["RequestMemory"].eval()
    1L
    >>> ad=classad.parse(open("test.submit.ad"))
    >>> print schedd.submit(ad, 2) # Submits two jobs in the cluster; edit test.submit.ad to preference.
    110
    >>> print schedd.act(condor.JobAction.Remove, ["111.0", "110.0"])'
    
        [
            TotalNotFound = 0;
            TotalPermissionDenied = 0;
            TotalAlreadyDone = 0;
            TotalJobAds = 2;
            TotalSuccess = 2;
            TotalChangedAds = 1;
            TotalBadStatus = 0;
            TotalError = 0
        ]
    >>> print schedd.act(condor.JobAction.Hold, "Owner =?= \"bbockelm\"")'
    
        [
            TotalNotFound = 0;
            TotalPermissionDenied = 0;
            TotalAlreadyDone = 0;
            TotalJobAds = 2;
            TotalSuccess = 2;
            TotalChangedAds = 1;
            TotalBadStatus = 0;
            TotalError = 0
        ]
    >>> schedd.edit('Owner =?= "bbockelm"', "Foo", classad.ExprTree('"baz"'))
    >>> schedd.edit(["110.0"], "Foo", '"bar"')
    >>> coll = condor.Collector()
    >>> master_ad = coll.locate(condor.DaemonTypes.Master)
    >>> condor.send_command(master_ad, condor.DaemonCommands.Reconfig) # Reconfigures the local master and all children
    >>> condor.version()
    '$CondorVersion: 7.9.4 Jan 02 2013 PRE-RELEASE-UWCS $'
    >>> condor.platform()
    '$CondorPlatform: X86_64-ScientificLinux_6.3 $'
    >>>