llama.pipeline module

Analysis Pipeline instances are Directed Acyclic Graphs (DAGs) specifying sets of FileHandler classes that together form an analysis pipeline. Use Pipeline instances to specify exactly which analysis steps you want to run; select a subset of the maximal DEFAULT_PIPELINE if you only need certain analysis outputs.

Pipelines

llama.pipeline.DEFAULT_PIPELINE : Pipeline

FileHandler dependency relations in llama.pipeline.DEFAULT_PIPELINE.

llama.pipeline.LLAMA2_REVIEW_PIPELINE : Pipeline

FileHandler dependency relations in llama.pipeline.LLAMA2_REVIEW_PIPELINE.

llama.pipeline.SUBTHRESHOLD_PIPELINE : Pipeline

FileHandler dependency relations in llama.pipeline.SUBTHRESHOLD_PIPELINE.

class llama.pipeline.FileHandlerSubgraphAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

Downselect for a subgraph on the specified filehandlers, returning a llama.pipeline.Pipeline containing the specified filehandlers and all of their ancestors if the option string starts with + or just the specified filehandlers if the option_string starts with -; combine results so that both types of flags can be used in the same command-line invocation.

class llama.pipeline.Parsers

Bases: object

A collection of CLI parsers implementing pipeline related functionality. Note that you will need to specify ``prefix_chars=”-+”`` to use the ``++filehandlers`` functionality. See llama.cli.Parsers for more information.

pipeline = CliParser(prog='sphinx-build', usage=None, description=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=False)

class llama.pipeline.Pipeline

Bases: llama.classes.ImmutableDict, llama.classes.NamespaceMappable

A pipeline specifies a specific set of data inputs and the functions that act on them in terms of intermediate data products and the functions used to generate them in a Directed Acyclic Graph (DAG); these products are bundled into FileHandlers. FileHandlers are graph nodes with DEPENDENCIES (edges) specified. A Pipeline DAG can be built purely by specifying the specific FileHandlers which can be done trivially and clearly at the file-system level by putting the FileHandler code into a single directory for each pipeline.

Parameters

• kwargs (dict) – Names of FileHandler classes mapped to the classes themselves.

• args (array-like) – FileHandler classes. The __name__ of each FileHandler will be used as the key.

Returns

pipeline – A new Pipeline instance containing all of the FileHandler classes specified in args and kwargs.

Return type

Pipeline

Raises

TypeError – If there are any name collisions between classes in the input args and kwargs, if any of the FileHandler classes it contains are abstract (non-implemented) classes, or if any of the FileHandler classes it contains have missing required_attributes.

check_consistency(other)

Check whether two Pipeline instances use the same keys to describe the same FileHandler classes, raising a ValueError if they don’t.

dependency_graph(outfile: str = None, title: str = 'Pipeline', url: function = None, bgcolor: str = 'black')

Return a graphviz .dot graph of DEPENDENCIES between file handlers in this pipeline. Optionally plot the graph to an output image file visualizing the graph.

Optional file extensions for outfile:

• dot: just save the dotfile in .dot format.

• png: save the image in PNG format.

• pdf: save the image in PDF format.

• svg: save the image in svg format.

Parameters

• outfile (str, optional) – If not provided, return a string in .dot file format specifying graph relationsIf an output file is specified, infer the filetype and write to that file.

• title (str, optional) – The title of the pipeline graph plot.

• url (FunctionType, optional) – A function taking FileHandler classes as input and returning a URL that will be added to each FileHandler class’s node in the output graph. Allows you to add links. If not included, URLs will not be included.

• bgcolor (str, optional) – The background color to use for the generated plot.

Returns

dot – The dependency graph in .dot format (can be used as input to dot at the command line). This is returned regardless of whether an outfile is specified.

Return type

str

downselect(invert=False, reducer=<built-in function all>, **kwargs)

Return a Pipeline instance whose FileHandler classes match ALL the given query parameters.

Parameters

• invert (bool, optional) – Invert results. (Default: False)

• reducer (function, optional) – Specify any builtin to match if any check passes. Specify all to match only when every check passes. (Default: all)

• type (type, optional) – The type of the FileHandler must exactly match the given FileHandler.

• typename (str, optional) – The FileHandler type’s name must match this string.

• subclass (type, optional) – The FileHandler must be a subclass of this FileHandler.

• subgraph (type, optional) – The FileHandler must be either this FileHandler or one of its UR_DEPENDENCIES; use this to make a Pipeline that only generates the subgraph leading to this FileHandler.

file_handler_instances(*args, **kwargs)

Return a FileHandlerMap with FileHandler instances sharing the same initialization arguments, e.g. for FileHandler instances that all refer to the same event.

classmethod from_module(module)

Create a pipeline by extracting all FileHandler objects from a given submodule.

class llama.pipeline.PipelineAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

Set the pipeline from the available pipelines defined in llama.pipeline.

class llama.pipeline.PrintDefaultPipeline(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

Bases: argparse.Action

Print DEFAULT_PIPELINE in a readable format and quit.

llama.pipeline.postprocess_pipeline_dry_run(self: llama.cli.CliParser, namespace: argparse.Namespace)

Print the filehandlers in namespace.pipeline and quit without evaluating further if namespace.dry_run_pipeline is True.

llama.pipeline.postprocess_pipeline_selection(self: llama.cli.CliParser, namespace: argparse.Namespace)

Add any filehandlers defined in namespace.filehandlers to namespace.pipeline. If namespace.pipeline is not defined, set it to the DEFAULT_PIPELINE.