llama.utils module¶

Utility functions for LLAMA with no dependencies on other LLAMA tooling.

class llama.utils.ColorFormatter(fmt=None, datefmt=None, style='%')¶

Bases: logging.Formatter

A formatter that colors log output for easier reading.

COLORS = {'CRITICAL': '\x1b[91m', 'DEBUG': '\x1b[94m', 'ERROR': '\x1b[93m', 'INFO': '\x1b[92m', 'WARNING': '\x1b[95m'}¶

format(record)¶

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class llama.utils.RemoteFileCacher¶

Bases: llama.utils.RemoteFileCacherTuple

A simple loader for remotely-cached data accessible on public URLs. Use this to download cloud-based resources and locally cache them in /root/.cache/llama/objects. Files will only be downloaded as-needed to save space and bandwidth.

Parameters

url (str) – Specify the URL of the remote resource.
localpath (str, optional) – The (optional) local path at which to cache this resource. By default, will just be /root/.cache/llama/objects/filename where filename is actually taken from the remote URL filename.

get(query=None)¶: If the file is not available locally, download it and store it at localpath (do nothing if present). Return localpath. Optionally append a query string query.

class llama.utils.RemoteFileCacherTuple(url, localpath)¶

Bases: tuple

property localpath¶: Alias for field number 1

property url¶: Alias for field number 0

llama.utils.archive_figs(plots, outfile, exts=('pdf', 'png'), fname_list=None, close_figs=False)¶

Take a list of figures and save them with filenames that look like {fname}.{suffix} to a .tar.gz archive named outfile. The list_index will be the zero-padded index of the given plot in the plots list. Default filenames are generated if fname_list is not given. For instance, with no fname_list specified, the third plot in plots would have files named named plt_003.pdf and plt_003.png saved to outfile (assuming default kwargs values).

Parameters

plots (list) – A list of matplotlib figures to save to file.
outfile (str) – Filename of the output archive. Should end in .tar.gz since this will be a zipped tarball of figures.
exts (list, optional) – List of filename extensions to use. Each extension becomes the suffix in the above format string. Allows multiple image formats to be saved for each plot.
fname_list (list, optional) – A list of file names to use for each figure (sans file extensions). NOTE: If not provided, some default filenames are generated instead. These follow the filename pattern plt_{list_index}.{suffix}.
close_figs (bool, optional) – If True, clear the figures in plots after plotting is finished. This will help reduce memory usage if the plots are not going to be reused.

llama.utils.bytes2str(bytes_or_str)¶: Convert an object that could be a bytes object (in python3) to a str object. If it is already a str object, leave it alone. Used for python2/python3 compatibility.

llama.utils.color(fg=None, bg=None)¶

Return full-color escape codes for changing foreground and background in terminals with RGB color support.

Parameters

fg (str or list, optional) – The foreground (text) color. A 3-item list containing integer values for red, green, and blue or a hexadecimal representation of the color prefixed by an octothorpe (like in CSS). Background will not be set if not specified.
bg (str or list, optional) – The background color. Same type as fg.

Returns

esc – The teriminal escape code to use.

Return type

str

Raises

ValueError – If fg or bg are not as specified.

llama.utils.color_logger(loglevel=None, outfile=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)¶: Return a function log that prints formatted strings to either sys.stdout or with the logger using a certain log level (if loglevel is specified). If loglevel is specified, then outfile is ignored.

llama.utils.find_in_submodules(modulename: str, predicate: function)¶: Return a dict of fully-qualified variable names mapping to all objects in modulename and its submodules for which predicate is True.

llama.utils.getVOEventGPSNanoseconds(voeventpath)¶: Get the number of nanoseconds past the start of the current GPS second of the event described in this VOEvent file. For example, if the current GPS time is 123456789.033, this method will return 33000000.

llama.utils.getVOEventGPSSeconds(voeventpath)¶: Get the GPS time of the event described in this VOEvent file to the nearest second.

llama.utils.get_grid(ra, dec, radius, pixels=100, degrees=True)¶

Get a list of pixel positions arranged in a rectangular grid and filling a square with sizes of length radius centered at Right Ascension ra and Declination dec with pixels as the width and height in pixels of the square grid.

This technically only works properly if the radius is very small, so a ValueError will be raised for radii larger than MAX_GRID_SCALE_RAD. For larger sky areas, HEALPix pixelizations should be used to assure equal area.

Parameters

ra (float) – The Right Ascension of the center of the grid.
dec (float) – The Declination of the center of the grid.
radius (float) – The radius of the circle. Must be larger than MAX_GRID_SCALE_RAD (in radians) to keep pixel areas nearly constant.
pixels (float, optional) – The diameter, in pixels, of the circle along the cardinal directions of the grid. Higher values imply higher resolutions for the grid.
degrees (bool, optional) – If true, interpret all input and returned angles as being measured in degrees. Otherwise, interpret them all as being measured in radians.

Returns

ras (array) – Right Ascension values of the grid pixels in the specified units.
decs (array) – Declination values of the grid pixels in the specified units.
area (float) – the (approximate) solid-angle per-pixel in the specified units.

Raises

ValueError – Raised when the radius is too large (larger than MAX_GRID_SCALE_RAD, see note above).

llama.utils.get_test_file(filename, eventid)¶

Get the path to an example file as would be generated by LLAMA. These are stored in a single directory (equivalent to an event directory) and can be used for unit, integration, and doc tests.

Parameters

filename (str) – The name of the file as it would appear in an event directory with no path information before it.
eventid (str) – The name of the event directory we will be using for this unit test. Since different test cases apply to different file handlers, you’ll need to make sure to avoid specifying an eventid that this filehandler is not meant to be tested on (whatever the reason).

Returns

test_filepath – The full path to the file to be used for testing.

Return type

str

llama.utils.get_voevent_gps_time(voeventpath)¶: Get the GPS time of the event described in this VOEvent file. Maximum precision is in nanoseconds, though in practice it will not be this high.

llama.utils.get_voevent_notice_time(voeventpath)¶: Get a unicode string with the date and time of the notification rather than the event itself. Is in ISO format.

llama.utils.get_voevent_param(voeventpath, param_name)¶

Get a ‘What’ parameter from a VOEvent XML file and return it as a string (no clever type inference is performed). For example, get the FAR (False Alarm Rate) for event G298048 (first BNS detection), which is included as a standard test case in ‘llama/data/tests/voevents’:

Examples

>>> get_voevent_param(get_test_file('lvc_gcn.xml',
...                                 'MS181101ab-2-Initial'), 'FAR')
'9.11069936486e-14'

Note that the type of the returned data will (probably) be a unicode string, so be sure to convert as necessary.

llama.utils.get_voevent_role(voeventpath)¶: Get the role of this VOEvent, test or observation.

llama.utils.get_voevent_time(voeventpath)¶

Get a unicode string with the ISO date and time of the event:

Examples

>>> get_voevent_time(get_test_file('lvc_gcn.xml',
...                                'MS181101ab-2-Initial'))
'2018-11-01T22:22:46.654437'

llama.utils.gps2mjd(gps)¶: Convert GPS seconds to MJD time

llama.utils.gps2utc(gps)¶: Convert GPS seconds to UTC string

llama.utils.is_main_process()¶: Return whether this is the main process (as opposed to a worker process).

llama.utils.label_worker_proc()¶: Label the current process as a worker process (not the main process). You should only call this as the initializer to an executor.

llama.utils.memoize(method=False)¶

A decorator that takes a method and memoizes it by storing the value in a class or instance variable. Useful when the same arguments are likely to keep recurring and the computation time is long. Data is stored in the function (for regular functions and staticmethods), the class (for classmethods) or in the instance (for properties and methods), this approach can’t work for staticmethods nor bare functions. For this to work, all arguments (except self or cls) must be hashable.

NOTE: all arguments must be hashable (with the exception of the class or instance when method is True; see below).

Parameters: method (bool, default=False) – If true, treat this function like a method, so that the first argument is a class or instance, and store the cache with that class or instance instead of storing it with the function. This lets you use function memoization on properties, classmethods, and instance methods even if the class or instance is not hashable.

llama.utils.mjd2gps(mjd)¶: Convert MJD time to GPS seconds

llama.utils.mjd2utc(mjd)¶: Convert MJD time to UTC string

llama.utils.parameter_factory(classname, description, **kwargs)¶

Return a class that can be used as a namespace for a bunch of parameters related to a specific search. Very similar to defining a namedtuple except that each parameter is defined as a property with a docstring, making these objects suitable for interactive use without having to open the source code to see parameter descriptions. Parameter names are provided as keyword argument names (see below).

The main point of this factory is that it provides a compact way of defining a new hashable class with descriptive docstrings; these features are good for classes representing collections of parameters (hence the function name).

The reason to implement this as a factory function rather than an abstract superclass is that the returned class has static attributes with docstrings available in a way that ipython can parse, making for easier interactive use of the built-in documentation.

Parameters

classname (str) – The name of the new class.
description (str) – A description of what sort of analysis these parameters are to be used for. Becomes part of the docstring of the new class (along with the descriptions included in parameters).
kwargs (str or function) – The names and descriptions (or, for derived values, formulas) of parameters to be stored. The names of the arguments become the parameter names; these will be attributes of the returned class. The values of the arguments can be strings serving as descriptions of the parameter (for INDEPENDENT parameters provided by the user, in which case the descriptive string becomes the docstring for the parameter in the returned class) or functions that take self as their only argument where self is an instance of the new parameter class and some of the attributes of said instance are used to calculate the value of this (DEPENDENT) parameter; in these cases, the docstring of the function is reused for that parameter. These dependent parameters become properties of the new class and are calculated dynamically from the class’s other values (possibly including other dependent parameters as long as there are no circular method calls).

Returns

new_class – a class whose name is classname, whose properties are all the parameter names, whose docstrings are their descriptions, whose docstring is the class description combined with the names and descriptions in the kwargs parameters, and whose __init__ signature requires that the parameters be passed as kwargs for the sake of explicitness in instantiation.

Return type

type

Examples

>>> def buz(self):
...     "bar+baz"
...     return self.bar + self.baz
>>> def boz(self):
...     "double buz"
...     return 2*self.buz
>>> Foo = parameter_factory(
...     "Foo",
...     "A silly example set of parameters.",
...     bar='a fake param',
...     baz='another fake param',
...     buz=buz,
...     boz=boz,
... )
>>> Foo.bar.__doc__
'(independent)\na fake param'
>>> Foo.baz.__doc__
'(independent)\nanother fake param'
>>> Foo.buz.__doc__
'(dependent)\nbar+baz'
>>> Foo.boz.__doc__
'(dependent)\ndouble buz'
>>> quux = Foo(bar=1, baz=2)
>>> quux.bar
1
>>> quux.baz
2
>>> quux.buz
3
>>> quux.boz
6

llama.utils.plot_graphviz(dot, outfile)¶

Plot a .dot graph (graph here used in the mathematical sense) and save it to outfile. Calls graphviz dot executable, used for plotting .dot graphs, so graphviz must be installed on the system.

Parameters

dot (str) – Graphviz .dot format graph (the same contents you would expect in a .dot file).
outfile (str) – Output file to save graph to. File type is inferred from outfile’s extension. This file type must be one of the ones recognized in GRAPH_EXTENSIONS.

Raises

subprocess.CalledProcessError – If the dot process fails to generate the output file. This exception object will have the nonzero return code in it.

llama.utils.rotate_angs2angs(ra, dec, yrot, zrot, degrees=True)¶

Take a list of angular sky positions (points on the sphere) and rotate them first through an angle yrot about the y-axis (right-handed) followed by an angle zrot about the z-axis. Return the new points as (ra, dec) coordinates of the rotated vectors.

Parameters

ra (array-like) – A list or numpy.ndarray of Right Ascension values for each point on the sphere.
dec (array-like) – A list or numpy.ndarray of Declination values for each point on the sphere. Must have same length as ra.
yrot (float) – The angle through which to do a right-handed rotation about the positive y-axis. This rotation is applied first.
zrot (float) – The angle through which to do a right-handed rotation about the positive z-axis. This rotation is applied second.
degrees (bool, optional) – (DEFAULT: True) If true, interpret all input angles as being measured in degrees and return values in degrees. Otherwise, interpret them as being measured in radians and return values in radians.

Returns

outra (array) – The Right Ascensions of the rotated input vectors in the specified units.
outdec (array) – The Declinations of the rotated input vectors in the specified units.

Examples

>>> import numpy as np
>>> def eql(a, b, prec=1e-13, mod=360):  # compares floating point arrays
...     a = np.array(a).flatten()
...     b = np.array(b).flatten()
...     return all(prec > (a%mod - b%mod))
>>> ra = range(360)
>>> dec = np.zeros(360)
>>> eql(rotate_angs2angs(ra, dec, 0, 0), [ra, dec])
True

llama.utils.rotate_angs2vec(ra, dec, yrot, zrot, degrees=True)¶

Take a list of angular sky positions (points on the sphere) and rotate them first through an angle yrot about the y-axis (right-handed) followed by an angle zrot about the z-axis. Return the new points as (x, y, z) coordinates of the rotated directions as vectors on the unit sphere.

Parameters

ra (array-like) – A list or numpy.ndarray of Right Ascension values for each point on the sphere.
dec (array-like) – A list or numpy.ndarray of Declination values for each point on the sphere. Must have same length as ra.
yrot (float) – The angle through which to do a right-handed rotation about the positive y-axis. This rotation is applied first.
zrot (float) – The angle through which to do a right-handed rotation about the positive z-axis. This rotation is applied second.
degrees (bool, optional) – (DEFAULT: True) If true, interpret all input angles as being measured in degrees. Otherwise, interpret them as being measured in radians.

Returns

x (array) – The x-coordinates of the rotated input vectors [cartesian].
y (array) – The y-coordinates of the rotated input vectors [cartesian].
z (array) – The z-coordinates of the rotated input vectors [cartesian].

Examples

>>> import numpy as np
>>> def eql(a, b, prec=1e-15):  # compares floating point arrays
...     return all(prec > np.array(a).flatten() - np.array(b).flatten())
>>> eql(rotate_angs2vec(0, 0, 0, 0), [1, 0, 0])
True
>>> eql(rotate_angs2vec(270, 0, 0, 0), [0, -1, 0])
True
>>> eql(rotate_angs2vec(270, 0, 0, 90), [1, 0, 0])
True
>>> eql(rotate_angs2vec(180, 0, 90, 90), [0, 0, 1])
True
>>> eql(rotate_angs2vec(0, 90, 90, 90), [0, 1, 0])
True
>>> eql(rotate_angs2vec(np.pi/2, 0, 0, np.pi, degrees=False), [0, -1, 0])
True
>>> eql(rotate_angs2vec(np.pi, 0, np.pi/2, 0, degrees=False), [0, 0, 1])
True

llama.utils.send_email(subject, recipients, body='', attachments=())¶

Send an email using the default configured email address. No return value.

Parameters

recipients (list) – A list of strings representing email addresses of the recipients.
body (str or file-like, optional) – Send contents of this string or file-like object.
attachments (str or list, optional) – A single filepath or a list of filepaths to files that should also be attached to this email.

llama.utils.setup_logger(logfile, loglevel='DEBUG')¶

Set up a logger for llama and all submodules that logs to both logfile and to STDOUT. You will still need to actually make a logger for whatever module you are calling this from.

Parameters

logfile (str) – The logfile to write all output to. If the path equals or resolves to /dev/null, no logfile will be configured.
loglevel (int, str, or NoneType, optional) – The loglevel to pass to setLevel for the StreamHandler writing to the terminal; the logfile handler always writes at maximum verbosity (DEBUG).

llama.utils.sizeof_fmt(num, suffix='B')¶: Format the size of a file in a human-readable way. num is the file size, presumed to be in bytes. Specify suffix='b' to indicate that the unit of num is bits.

llama.utils.sort_files_by_modification_time(paths)¶: Take the list of path strings sorted by ascending modification time. This is the last time the contents were modified, or, for a new file, its creation time.

llama.utils.tbhighlight(tb)¶

Syntax-highlight a traceback for a 256-color terminal.

Parameters: tb (str) – The traceback string (as provided by traceback.format_exc())
Returns: highlighted_tb – The same traceback string with syntax highlighting applied.
Return type: str

llama.utils.utc2gps(utc)¶: Convert UTC time string to GPS seconds

llama.utils.utc2mjd(utc)¶: Convert UTC time string to MJD time

llama.utils.veccls(func)¶: Like vecstr but for class arguments instead of string arguments.

llama.utils.vecfh(func)¶: Like vecstr but for FileHandler instances.

llama.utils.vecstr(func)¶: Given a function taking a FileHandler instance fh and a query argument and returning a bool, return a function with the same signature that first checks whether query is a str before calling func. If query is a str, return func(fh, query), and if it is not, return any(func(fh, q) for q in query), i.e. check whether func is true for any of the values given in query. Note that this means query can either be a check against a str or against several possible values of str contained in an iterable. This function is an implementation detail used for downselection checks.

llama.utils.vectypes(func, types)¶: Implementation of vecstr and veccls.

llama.utils.write_gzip(infile, outfilename)¶

See whether a file is actually a valid gzip file. If not, zip it. Either way, write the result to outfilename.

Parameters

infile (file-like) – File-like object to be compressed to a gzip file.
outfilename (str) – Path to the output compressed file. Will be overwritten if it already exists.

Returns

success – True if it was already a valid gzip file and False if we were forced to compress it.

Return type

bool

Examples

Compress a string in a file-like BytesIO wrapper to some temporary .gz file: >>> from io import BytesIO >>> import tempfile >>> with tempfile.NamedTemporaryFile(suffix=’.gz’, delete=False) as f: … outfilename = f.name >>> infile = BytesIO(‘this is obviously not a valid gzip file’.encode()) >>> assert not write_gzip(infile, outfilename) >>> with open(outfilename, ‘rb’) as actuallyzippedfile: … assert write_gzip(actuallyzippedfile, outfilename)

llama.utils.write_to_zip(fname_list, write_functions, outfilename, suffix='')¶

Create a new .tar.gz tarfile archive names outfilename and fill it with filenames given in fname_list. write_functions should be a list of functions that takes an input filename as its only argument and writes data to that filename. It should have the same length as fname_list, since each element of write_functions will be used to generate the corresponding file in fname_list. If the write_functions expect some sort of specific file extension, that can be provided by manually setting suffix (DEFAULT: ‘’) to the appropriate extension. If each file requires its own specific filename extension, then suffix can be specified as a list of suffixes with length matching fname_list and write_functions.

Examples

>>> import tempfile, tarfile
>>> def foo(file):
...     with open(file, 'w') as f:
...         f.write('foo!')
>>> write_functions = [foo] * 3
>>> fname_list = ['bar.txt', 'baz.txt', 'quux.txt']
>>> outfile = tempfile.NamedTemporaryFile(delete=False)
>>> outfilename = outfile.name
>>> outfile.close()
>>> write_to_zip(fname_list, write_functions, outfilename, suffix='.txt')
>>> tar = tarfile.open(outfilename)
>>> [f.name for f in tar.getmembers()]
['bar.txt', 'baz.txt', 'quux.txt']
>>> os.remove(outfilename)