Experimental Panda Monitor

This documentation is for EXPERIMENTAL features.

Both the feature and the documentation are subject to change without notice.

Introduction

The PanDA monitor forwards job ClassAd changes from a schedd to a PanDA monitor's RESTful update interface. Enabling this feature will add some overhead to the schedd, and an additional process to its machines (where most of the work will be done).

This experimental feature consists of two parts: a schedd plug-in and a separate process, the panda daemon. The former streams updates (without blocking) to the latter, which buffers the stream and periodically sends batched commands (add, update, remove) to the RESTful interface. Updates will be dropped if the daemon's buffers fill up.

Configuration

This experimental feature is enabled by setting SCHEDD.PLUGIN_DIR or SCHEDD.PLUGINS appropriately. In many cases, setting SCHEDD.PLUGIN_DIR to $(LIBEXEC) will work, but it's safer to set SCHEDD.PLUGINS to include the plugin binary (panda-plugin.so).

The following is a list of configuration settings.

PANDA_URL A URL. Where's the RESTful interface? No default.

PANDAD A path. Points to the pandad binary and defaults to $(LIBEXEC)/pandad.

PANDA_UPDATE_TIMEOUT An integer, in seconds. How long (forever, if 0) should the panda daemon wait for any particular update to complete before giving up? (Which causes updates to be lost.) Defaults to 0.

PANDA_QUEUE_SIZE An integer, in bytes. How large of a buffer should the panda daemon provide? Defaults to 131072. Updates may be lost if the buffer is too small.

PANDA_QUEUE_GRACE An integer, in seconds. For how long, in seconds, should the panda daemon continue to accept entries into the buffer after it exceeds PANDA_QUEUE_SIZE? Useful for dealing with bursty traffic. Defaults to 3.

PANDA_STATISTICS_LOG A path. Points to a file on disk where you'd like the panda daemon to occasionally write statistics about its performance. Mostly for debugging. Defaults to '/tmp/pandaStatisticsLog'.

PANDA_JOB_ATTRIBUTES A space-separated list. If you only want to send a subset of job attributes, list those attributes in this configuration macro. No default, but if the macro is unset or empty, the daemon sends all attributes.

Random Thoughts

If you set PANDA_QUEUE_GRACE to one more than PANDA_UPDATE_TIMEOUT, you should only lose that single timed-out connection's updates.

Updates happen no more than once a second, although on an idle schedd, they may happen less often.

By "updates", the previous sentence means "one ADD command, one UPDATE command, and one REMOVE command" (in that order). This is probably true elsewhere on this page.