Condor In The Cloud Seed Construction

Condor in the Cloud Seed Construction

The following is internal documentation.


(Instructions assume root.)

  • Start the standard RHEL7.6 image.
  • Install wget.
    yum install wget
  • Follow the instructions here .
  • Grab the condor-annex-ec2 script:
    yum install condor-annex-ec2
  • Install the AWS CLI tool needed by that script:
    $ yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
    $ yum install python-pip
    $ pip install --upgrade pip
    $ pip install awscli
    
  • Rather than store your AWS tokens in the AMI, if you didn't start the instance with a role that can run condor_annex , add one now (via the EC2 web console).
  • Configure HTCondor:
    /etc/condor/config.d/local
    # Make this a single-node pool.
    DAEMON_LIST = MASTER COLLECTOR SCHEDD STARTD NEGOTIATOR
    
    # Taken from the manual's instructions on installing a minicondor.
    # ------------------------------------------------------------------------------
    
    NETWORK_INTERFACE = *
    CONDOR_HOST = $(FULL_HOSTNAME)
    
    SEC_DAEMON_INTEGRITY = REQUIRED
    SEC_DAEMON_AUTHENTICATION = REQUIRED
    SEC_DAEMON_AUTHENTICATION_METHODS = PASSWORD
    SEC_NEGOTIATOR_INTEGRITY = REQUIRED
    SEC_NEGOTIATOR_AUTHENTICATION = REQUIRED
    SEC_NEGOTIATOR_AUTHENTICATION_METHODS = PASSWORD
    SEC_CLIENT_AUTHENTICATION_METHODS = FS, PASSWORD
    ALLOW_DAEMON = condor_pool@*
    
    # ------------------------------------------------------------------------------
    # The following is different from the instructions because the RHEL7 RPM
    # sets $(LOCAL_DIR) in a deeply unfortunate way.
    SEC_PASSWORD_FILE = /etc/condor/condor_pool_password
    
    # Configure the pool to be externally accessible.  While this isn't necessary
    # for the proper functioning of an AWS-only pool, condor_annex can't verify
    # the accessibility of this host if it's not externally available, and there's
    # not presently an option to disable that check.
    #
    # Note that EC2PublicIP is set by the condor-annex-ec2 package.
    TCP_FORWARDING_HOST = $(EC2PublicIP)
    COLLECTOR_HOST = $(TCP_FORWARDING_HOST):9618
    ALLOW_WRITE = $(ALLOW_WRITE) $(TCP_FORWARDING_HOST)
    ALLOW_NEGOTIATOR = $(ALLOW_NEGOTIATOR) $(TCP_FORWARDING_HOST)
    
  • Enable the condor-annex-ec2 service.
    systemctl enable condor-annex-ec2
  • rm /etc/condor/config.d/50ec2.config
  • Edit /usr/libexec/condor/condor-annex-ec2 so that it only sets EC2PublicIP and EC2InstanceID : remove all the lines from "Determine the annex ID" (60) to where the redirect to /etc/condor/config.d/49ec2-instance.config appears (342). (Do not delete the redirect line.) Replace the next line's '$?' with '0'.
  • Either start the condor-annex-ec2 service now, and verify that /etc/condor/config.d/49ec2.config exists and is correct, or reboot at the end of the instructions (and then verify the file). The configuration above does not work, otherwise.
  • Run
    condor_store_cred -c add -f `condor_config_val SEC_PASSWORD_FILE`
    as root to create the pool password file.
  • Make sure the password file (run condor_config_val SEC_PASSWORD_FILE to find it) is owned by root and has 600 permissions afterwards.
  • Then copy the pool password file to ~ec2-user/.condor and chown it to that user.
  • Edit ~ec2-user/.condor/user_config and add the line SEC_PASSWORD_FILE = /home/ec2-user/.condor/condor_pool_password ; this allows condor_annex to copy the pool password file to the new instances.


You should now have an Condor-in-the-Cloud seed instance. Convert to an AMI in the usual way.

The following, if copied and pasted into a terminal window just before you hit 'create image', makes the image more suitable for general usage: it turns off bash history, clears the bash history, and removes the bash history file; it also removes (all?) HTCondor history files, so the AMI's HTCondor starts up with a clean state; and then cleans up the root and current user's SSH keys and known hosts, and also root's bash history.

set +o history
history -c
rm -fr ~/.bash_history
sudo rm -fr /var/log/condor/*
sudo rm -fr /var/run/condor/*
sudo rm -fr /var/lock/condor/InstanceLock
sudo rm -fr /var/lib/condor/execute/*
sudo rm -fr /var/lib/condor/spool/*
sudo rm -fr /etc/condor/config.d/49ec2-instance.config
sudo rm -fr /etc/condor/config.d/password_file.pl
sudo /bin/sh -c 'rm -fr ~/.bash_history'
sudo /bin/sh -c 'rm -fr ~/.ssh/authorized_keys'
sudo /bin/sh -c 'rm -fr ~/.ssh/known_hosts'
rm -fr ~/.ssh/authorized_keys
rm -fr ~/.ssh/known_hosts