llama run

The main interface for running the LLAMA pipeline. Turns on the automated pipeline, either in the foreground (for immediate work) or in the background (launching a long-running daemon process).

The pipeline checks for new LLAMA events and event data and (re)fetches/(re)generates data files for each event as new data becomes available. Provides a very flexible and powerful command line interface for running LLAMA.

usage: llama run [-h] [-D] [-f [FILEHANDLER [FILEHANDLER ...]]]
                 [+f [FILEHANDLER [FILEHANDLER ...]]]
                 [-p {DEFAULT_PIPELINE,LLAMA2_REVIEW_PIPELINE}]
                 [--print-default-pipeline] [--dry-run-pipeline]
                 [--dry-run-dirs] [-l LOGFILE]
                 [-v {debug,info,warning,error,critical,none}] [-o]
                 [-n NUM_UPDATED] [--delay DELAY] [-F] [-R] [-k] [-r]
                 [run]

Named Arguments

-D, --dev-mode
If specified, allow the program to run even if the source directory is not clean. You should not do this during production except in an emergency. If the flag is not specified but local changes to the source code exist, llama run will complain and quit immediately (the default behavior).

Default: False

-o, --runonce
If specified, each event directory will only be updated once, and then the script will exit.

Default: False

-n, --num-updated
 
Number of event directories to keep updated. Older event directories will be skipped and not updated if they do not fall within the top num_updated.

Default: 40

--delay
A delay time between generation attempts. Useful if generating many files pulling from an API to avoid DOSing someone.
-F, --foreground
 
Run llama run in the foreground instead of immediately sending it to the background as a daemon process. If running in the foreground, logs are printed to STDERR (in addition to the global archival logfile, see --logfile); if running in the background (i.e. omitting this flag), logs are appended to a logfile specific to this set of event directories (again, in addition to the archival logfile). This CLI-argument-specific logfile goes in the same log directory as the default --logfile option but has a filename specific to this llama run invocation; the name is the same as the lockdir name (see: lockdir), but with .log appended.

Default: False

-R, --running-daemons
 
Print the currently-running llama daemons as recorded in the llama run lockfile directory, /home/vagrant/.cache/llama/llamad.run.d. Note that this will not catch llama run processes whose lock files have somehow been deleted from this directory. Run ps -ax | grep 'llama run' if you’re paranoid about missing anything.
-k, --kill
Kill any llama processes covering potentially competing run and eventidfilter values (as decided by conservative comparison algorithms) by finding their PIDs in their semaphore lock directories and killing the associated processes (if they are still running). You can run this if a llama run instance fails to launch due to competing instances in order to kill them safely.

Default: False

-r, --relaunch
If relaunch is specified, then wrap llama run as a subprocess and keep relaunching it if it fails; if it exits gracefully (returncode 0), it will not be relaunched. All command line arguments (besides this one) will be passed to subprocess llama run invocations.

Default: False

choose pipeline (see ``llama.pipeline``)

-f, --filehandlers
 

Possible choices: LvcDistancesJson, CoincScatterI3LvcPng, CoincScatterZtfLVCPdf, LVAlertAdvok, RctSlkLmaLVCInitialXml, RctSlkLmaCoincScatterZtfLVCPdf, IceCubeNeutrinoListTxt, CoincSummaryI3LvcTex, I3LvcGcnDraft, RctSlkI3CoincSummaryI3LvcPdf, Advok, CoincScatterI3LvcPdf, LVCGraceDbEventData, RctSlkLmaCoincSummaryI3LvcPdf, LVCInitialXml, IceCubeNeutrinoListCoincTxt, SkymapInfo, RctSlkLmaCoincScatterI3LvcPdf, RctSlkLmaSkymapInfo, ZtfTriggerList, RctSlkLmaCoincScatterI3LvcPng, RctSlkLmaLVCPreliminaryXml, RctSlkLmaLVCGraceDbEventData, LvcSkymapFits, LvcSkymapHdf5, LVCPreliminaryXml, PAstro, RctSlkLmaLvcDistancesJson, FermiGRBsJSON, CoincScatterZtfI3LvcPdf, IceCubeNeutrinoList, RctSlkLmaCoincScatterZtfLVCPng, RctSlkLmaCoincScatterZtfI3LvcPng, CoincScatterZtfLVCPng, RctSlkLmaLVAlertJSON, RctSlkLmaCoincScatterZtfI3LvcPdf, CoincSummaryI3LvcPdf, CoincScatterZtfI3LvcPng, LVAlertJSON, IceCubeNeutrinoListTex, RctSlkLmaCoincSignificanceI3Lvc, CoincSignificanceI3Lvc

A list of FileHandler class names, separated by commas, which should be used. If provided, FileHandler classes whose names are in this list will be included in the pipeline. If the dependencies of a requested file are not available, no attempt will be made to generate them unless they too are listed explicitly. Available filehandlers are drawn from the DEFAULT_PIPELINE (print them with --print-default-pipeline).
+f, ++filehandlers
 

Possible choices: LvcDistancesJson, CoincScatterI3LvcPng, CoincScatterZtfLVCPdf, LVAlertAdvok, RctSlkLmaLVCInitialXml, RctSlkLmaCoincScatterZtfLVCPdf, IceCubeNeutrinoListTxt, CoincSummaryI3LvcTex, I3LvcGcnDraft, RctSlkI3CoincSummaryI3LvcPdf, Advok, CoincScatterI3LvcPdf, LVCGraceDbEventData, RctSlkLmaCoincSummaryI3LvcPdf, LVCInitialXml, IceCubeNeutrinoListCoincTxt, SkymapInfo, RctSlkLmaCoincScatterI3LvcPdf, RctSlkLmaSkymapInfo, ZtfTriggerList, RctSlkLmaCoincScatterI3LvcPng, RctSlkLmaLVCPreliminaryXml, RctSlkLmaLVCGraceDbEventData, LvcSkymapFits, LvcSkymapHdf5, LVCPreliminaryXml, PAstro, RctSlkLmaLvcDistancesJson, FermiGRBsJSON, CoincScatterZtfI3LvcPdf, IceCubeNeutrinoList, RctSlkLmaCoincScatterZtfLVCPng, RctSlkLmaCoincScatterZtfI3LvcPng, CoincScatterZtfLVCPng, RctSlkLmaLVAlertJSON, RctSlkLmaCoincScatterZtfI3LvcPdf, CoincSummaryI3LvcPdf, CoincScatterZtfI3LvcPng, LVAlertJSON, IceCubeNeutrinoListTex, RctSlkLmaCoincSignificanceI3Lvc, CoincSignificanceI3Lvc

Exact same as --filehandlers, but all ancestors of the files listed with the + prefix will also be included. This means that, if you ask to generate a single file, an attempt will be made to also generate everything it depends on if necessary. If you want all those files made, this is a handy shortcut; if you want file generation to fail when ancestors are missing, use the - prefix instead.
-p, --pipeline

Possible choices: DEFAULT_PIPELINE, LLAMA2_REVIEW_PIPELINE

The name of the pipeline to use. Must be the name of a Pipeline instance from llama.pipeline. Available choices: [‘DEFAULT_PIPELINE’, ‘LLAMA2_REVIEW_PIPELINE’]. If both this and filehandlers are specified, then the resulting pipeline will include all requested filehandlers from both options.
--print-default-pipeline
 
Print the contents of the default pipeline and quit.
--dry-run-pipeline
 
Print the pipeline selected by the user and quit without taking further action. Use this to make sure you’ve specified the correct pipeline.

Default: False

filter runs and events (see: ``llama.run``)

run
A pattern specifying a list of directories to update of the form /run/directory/event*glob. See end of llama run -h documentation for details. (default: /home/vagrant/.local/share/llama/current_run/*
--dry-run-dirs
Print the runs and event directories that would be affected and exit without taking further action.

Default: False

logging settings

-l, --logfile
File where logs should be written. By default, all logging produced by llama run goes to both an archival logfile shared by all instances of the process as well as STDERR. The archival logfile can be overridden with this argument. If you specify /dev/null or a path that resolves to the same, logfile output will be suppressed automatically. Logs written to the logfile are always at maximum verbosity, i.e. DEBUG. (default: /home/vagrant/.local/share/llama/logs/llamad.log)

Default: “/home/vagrant/.local/share/llama/logs/llamad.log”

-v, --verbosity
 

Possible choices: debug, info, warning, error, critical, none

Set the verbosity level at which to log to STDOUT; the --logfile will ALWAYS receive maximum verbosity logs (unless it is completely supressed by writing to /dev/null). Available choices correspond to logging severity levels from the logging library, with the addition of none if you want to completely suppress logging to standard out. (default: info)

Default: “info”

Each LLAMA trigger gets its own directory. The name of this directory is called the eventid and the trigger itself is a LLAMA Event (see: llama.event). For a given LLAMA run, all event directories should go in a commond directory called a “run directory”; the collection of events is called a Run (see: llama.run). Most things the pipeline does work on a single Run and are meant to affect one or more matching Event instances. When you specify directories, you are implicitly specifying the Run (i.e. collection of triggers) as well as a UNIX-style glob (like the asterisk matching all files, *) which describes the eventid pattern you want to match. For example, matching all event IDs that start with “S” (corresponding to O3 LIGO/Virgo superevents) would require using S* as your event glob.

If you want to explicitly print which currently-existing Event directories will be impacted by the arguments you provide, you can use --dry-run-dirs to print the impacted directories and exit without taking further action. This is good practice while getting used to this interface.

The syntax for specifying the Run and Event glob is the path of the run directory followed by a slash followed by the event glob with no slash at the end (be sure to escape the * so the shell doesn’t expand it):

'/run/directory/event*glob'

Specify only the event glob by leaving the run directory out but keeping the leading / (if for some insane reason your root directory is your run directory, a double-leading / will communicate your perverse desire). In this case the default Run directory /home/vagrant/.local/share/llama/current_run/ is implied, so the following are equivalent:

'/event*glob'
/home/vagrant/.local/share/llama/current_run/'event*glob'

Specify only the Run directory by leaving a trailing slash and omitting the event glob; in this case, the default event glob * will be used, so the following are equivalent:

/run/directory/
/run/directory/'*'

You can use relative paths for the Run directory, the final part of the path will not be expanded and will be treated as the base directory. The only exception to this is if you are using relative paths and don’t put any / in the specified path, in which case the relative path will be expanded. This allows the common and intuitive behavior of running specific events in the current directory when you pass their name alone, or alternatively to treat the current directory as the only event directory by passing a single . as the run argument. Something like ./., however, will be interpreted as meaning you want the current directory to be the run directory only matching Event ids of ..

The following examples assume you are currently in the event directory /some/directory/. Let’s say this is the event directory, and you want to update only the contents of this directory. You can specify the run as /some/ and the event glob as directory with either of the following commands paths:

.
/some/directory

Alternatively, if /some/directory/ is a run directory, and you want to affect the event directories it contains that match the default event glob *, you can run use any of the following (note again that the event glob is in quotes to prevent your shell from expanding it into multiple arguments):

./
./'*'
/some/directory/
/some/directory/'*'

If you want to use the name of the current directory as your event glob (so that only eventids that have the same basename as your current directory are used) while keeping the default run directory /home/vagrant/.local/share/llama/current_run/, you would have to place a leading slash followed by the actual name of the run directory; as noted above, /. not work because the dot will be treated literally as the eventid you want to use. (Note that you usually wouldn’t want to do this; why would you be in this directory if you want to operate on an event stored in a different run directory?):

/directory
/home/vagrant/.local/share/llama/current_run/directory

See llama.run and llama.event for more information on Run and Event objects.

Note that by default, llama run will start a background process and exit from the foreground, allowing you to keep working or disconnect your SSH session while llama run progresses. Override this behaviour with -F.

Continually monitor and update the default run directory

llama run

Continually monitor and update a temporary test directory but NOT the default run directory

llama run /tmp/llamatest

Keep monitoring only the current directory

llama run .

Make a single attempt to generate IceCubeNeutrinoList in the current directory (if its input files exist) and then quit:

llama run . -o -f IceCubeNeutrinoList

Same as above, but generate any missing IceCubeNeutrinoList inputs if possible:

llama run . -o +f IceCubeNeutrinoList

Same as above, but instead of spinning up a background process, keep control of the terminal and print all logging output until the job is done:

llama run . -o +f IceCubeNeutrinoList -F