llama.batch package

Tools for analyzing large-scale batches of LLAMA data, e.g. to run simulations off of injected or randmoized data, with inputs and results stored locally or on a cloud storage service.

class llama.batch.DictAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

Split a list of command line arguments into an OrderedDict along the first occurence of “=” in each one.

llama.batch.batch_error_alert_maintainers(errdump, event, params)

Get an error callback function like llama.cli.traceback_alert_maintainers to pass to llama.cli.log_exceptions_and_recover that provides batch-specific information about the error.

Parameters

  • errdump (array-like) – A list of locations in which dumped data will go.

  • event (llama.event.Event) – The event which errored. Will be noted in the Slack alert message.

  • params (dict) – The dictionary that was used to format errdump containing values from --params. Contains information on the current event being processed.

llama.batch.batch_error_dump(errdump, event)

Get an error callback function like llama.cli.traceback_alert_maintainers to pass to llama.cli.log_exceptions_and_recover that dumps the results of the failing event to the directories listed in errdump.

Parameters

  • errdump (array-like) – A list of locations in which dumped data will go. These can be S3 “directory” paths compatible with put_file.

  • event (llama.event.Event) – The event which errored. Files will be dumped from here.

llama.batch.load_params(path: str)

Use read_file to read a local or remote path. Decode the file contents as either a JSON list or newline-delimited text file, and return the decoded list.

llama.batch.postprocess_param_iterator(self: llama.cli.CliParser, namespace: argparse.Namespace)

Overrides the namespace.params value with an iterator that captures all desired combinations of the specified params. Loads from file, HTTP, HTTPS, or S3 for each param list. If the value for namespace.random is not None, will instead become an infinitely long generator serving random combinations of the parameters.

Parameters

  • self (CliParser) – The CliParser applying this post-processing.

  • namespace (Namespace) – The processed arguments parsed by self.

Returns

combinations – A generator returning a namedtuple whose names are the keys of parameters and whose values are drawn from the corresponding value of parameters.

Return type

GeneratorType

llama.batch.put_file(source: str, dest: str, public: bool = False)

Save a file somewhere. Same as copying the file, but with S3 compatibility. The destination directory and all parent directories will be created if they don’t exist.

Parameters

  • source (str) – The path to the file to be uploaded.

  • dest (str) – Destination path. Works for local paths or for S3 keys (by prepending s3://{bucket}/ where by {bucket} is the name of the S3 bucket to which the file should be uploaded). Unlike read_file, does not work for uploading via HTTP/HTTPS. Intermediate directories will be made for local paths.

  • public (bool, optional) – Whether files uploaded to S3 should be publicly available. This will have no effect on local file destinations.

llama.batch.read_file(path: str)

Load data from a file path, either local, via HTTP/HTTPS, or from an S3 bucket.

Parameters

path (str) – Path to a JSON or (newline-delimited) text file containing a list of values to load. Path can also be an S3 object, denoted with “s3://” as the prefix, or a URL, denoted with “http://” or “https://” as the prefix. In these cases, the file is not saved locally.

Returns

data – The contents of the file.

Return type

bytes