llama.files.i3.json module

Methods and FileHandler classes associated with fetching IceCube neutrino triggers.

class llama.files.i3.json.IceCubeNeutrinoList

Bases: llama.filehandler.JSONFile, llama.files.healpix.skymap.HEALPixPSF

digraph "llama.files.i3.json.IceCubeNeutrinoList" { charset="utf-8" splines=ortho bgcolor=none margin=0 rankdir=LR "llama.files.i3.json.IceCubeNeutrinoList" [label=<{{<B>IceCubeNeutrinoList</B>|<I>icecube_neutrino_list.json</I>|JSONFile<BR/>HEALPixPSF}}>, shape="record", style=filled, target=_top, URL="/llama.files.i3.json.html#llama.files.i3.json.IceCubeNeutrinoList", fillcolor="#cccccc"]; "llama.files.skymap_info.SkymapInfo" [label=<{{<B>SkymapInfo</B>|<I>skymap_info.json</I>|EventTriggeredFileHandler<BR/>JSONFile}}>, shape="record", style=filled, target=_top, URL="/llama.files.skymap_info.html#llama.files.skymap_info.SkymapInfo", fillcolor="#cccccc"]; "llama.files.skymap_info.SkymapInfo" -> "llama.files.i3.json.IceCubeNeutrinoList" [arrowtail="none", color="red", ]; }

Required input files for llama.files.i3.json.IceCubeNeutrinoList (located on the far right of the graph). For a typical trigger, the leftmost files will be generated as triggers become available. They will be used as input for files to their right, eventually allowing llama.files.i3.json.IceCubeNeutrinoList to be generated.

Takes a list of dictionaries describing neutrinos and saves it to a text file. Validates the neutrino data to make sure each neutrino contains (minimally) the following properties:

  • mjd

  • zenith

  • azimuth

  • sigma

  • energy

  • type (one the values found in NEUTRINO_TYPES)

If this file is generated within the temporal search window for neutrinos, it is possible not all neutrinos needed for the final result will be available. In this case, the file will automatically become obsolete and will be regenerated after the window has elapsed.

Neutrinos from IceCube can be delayed due to communications issues; to combat this, when neutrinos are pulled from the online data stream after the temporal search window has elapsed, a check will be run to see if the next neutrino after the end of the search window is yet available. If it is not available, it is assumed that not all neutrinos needed for the search have arrived from south pole, and a GenerationError will be raised, allowing the pipeline to cooldown and retry the neutrino pull later.

Neutrinos pulled from before MJD_TOPIC_SWITCH_17_TO_BLANK will be automatically pulled from local GFU archives (stored online on LLAMA S3 and cached locally after use in llama.utils.CACHEDIR in *.npy format). Neutrinos from after this date will be pulled from the online GFU stream.

DEPENDENCIES = (<class 'llama.files.skymap_info.SkymapInfo'>,)
DETECTORS = (Detector(name='IceCube', abbrev='i3', fullname='IceCube', url='http://wiki.icecube.wisc.edu', summary='IceCube', description='', citations=ImmutableDict({})),)
FILENAME = 'icecube_neutrino_list.json'
MANIFEST_TYPES = (<class 'llama.files.i3.json.IceCubeNeutrinoList'>,)
UR_DEPENDENCIES = (<class 'llama.files.skymap_info.SkymapInfo'>,)
UR_DEPENDENCY_TREE = frozenset({<class 'llama.files.skymap_info.SkymapInfo'>})
downselect(ra=None, dec=None, ntype=None)

Downselect the neutrino list to match given criteria. Save the old neutrino list with the original filename prefixed by a timestamp and replace it with the downselected list.

Parameters

  • ra (list) – Right-ascension intervals (in degrees) that the neutrinos must fall into, e.g. [(21, 41),(92, 102)]

  • dec (list) – Decliination intervals (in degrees) that the neutrinos must fall into, e.g. [(-3, 20)]

  • ntype (str) – A type that the neutrino must match, e.g. 'observation' for real neutrinos.

end_of_neutrino_window()

The GPS time at which all neutrinos should be in under normal circumstances.

is_complete()

Whether the list of neutrinos is complete. Assumes incomplete if no complete key specified.

is_obsolete(checked=None, **kwargs)

IceCubeNeutrinoList files can be fetched immediately after a GW event becomes available. However, we want to eventually run our analysis on all neutrinos in a certain time window around an event. We therefore want to re-fetch the neutrinos in that time window after the time window has passed (allowing a safety factor for the IceCube GFU API to process new neutrino triggers). To do this, we mark the IceCubeNeutrinoList as obsolete if the current time is after the time window + safety margin has elapsed and the neutrino list was generated before that time had passed (meaning there might have been new neutrinos saved that are useful for the analysis after the file was originally generated). Standard FileHandler obsolescence criteria are also used to determine obsolescence through a call to super.

property neutrinos

Return a list containing a Neutrino object for each neutrino; more convenient to work with when writing formulae than dealing directly with the neutrino dictionaries.

property num_triggers

The number of triggers described by this file. Useful mostly for quickly determining if this trigger list is empty.

source_locations()

Returns a list of tuples of (RA, Dec, sigma) for all point-like sources, where (RA, Dec) is the central sky position and sigma is the standard deviation of the Gaussian PSF. Since this depends on the structure of data in the relevant file handler, it must be specified for each subclass.

property template_skymap_filehandler

The HEALPixSkyMapFileHandler whose HEALPix parameters should be used as a template for the HEALPixSkyMap output by this FileHandler. It is assumed that this HEALPixSkyMapFileHandler only returns one skymap in its list; consequently, only the first skymap loaded by this file handler will be used. Note that the template skymap file handler is likely not a dependency of this file handler; it is only used as a default for formatting generated skymaps from this file handler’s data.

class llama.files.i3.json.RctGdbIceCubeNeutrinoList

Bases: llama.files.gracedb.GraceDBReceipt

digraph "llama.files.i3.json.RctGdbIceCubeNeutrinoList" { charset="utf-8" splines=ortho bgcolor=none margin=0 rankdir=LR "llama.files.i3.json.IceCubeNeutrinoList" [label=<{{<B>IceCubeNeutrinoList</B>|<I>icecube_neutrino_list.json</I>|JSONFile<BR/>HEALPixPSF}}>, shape="record", style=filled, target=_top, URL="/llama.files.i3.json.html#llama.files.i3.json.IceCubeNeutrinoList", fillcolor="#cccccc"]; "llama.files.i3.json.RctGdbIceCubeNeutrinoList" [label=<{{<B>RctGdbIceCubeNeutrinoList</B>|<I>rct_gdb_icecube_neutrino_list.json.log</I>|GraceDBReceipt}}>, shape="record", style=filled, target=_top, URL="/llama.files.i3.json.html#llama.files.i3.json.RctGdbIceCubeNeutrinoList", fillcolor="#cccccc"]; "llama.files.skymap_info.SkymapInfo" [label=<{{<B>SkymapInfo</B>|<I>skymap_info.json</I>|EventTriggeredFileHandler<BR/>JSONFile}}>, shape="record", style=filled, target=_top, URL="/llama.files.skymap_info.html#llama.files.skymap_info.SkymapInfo", fillcolor="#cccccc"]; "llama.files.i3.json.IceCubeNeutrinoList" -> "llama.files.i3.json.RctGdbIceCubeNeutrinoList" [arrowtail="none", color="red", ]; "llama.files.skymap_info.SkymapInfo" -> "llama.files.i3.json.IceCubeNeutrinoList" [arrowtail="none", color="red", ]; }

Required input files for llama.files.i3.json.RctGdbIceCubeNeutrinoList (located on the far right of the graph). For a typical trigger, the leftmost files will be generated as triggers become available. They will be used as input for files to their right, eventually allowing llama.files.i3.json.RctGdbIceCubeNeutrinoList to be generated.

DEPENDENCIES = (<class 'llama.files.i3.json.IceCubeNeutrinoList'>,)
FILENAME = 'rct_gdb_icecube_neutrino_list.json.log'
MANIFEST_TYPES = (<class 'llama.files.i3.json.RctGdbIceCubeNeutrinoList'>,)
UPLOAD

alias of IceCubeNeutrinoList

UR_DEPENDENCIES = (<class 'llama.files.skymap_info.SkymapInfo'>, <class 'llama.files.i3.json.IceCubeNeutrinoList'>)
UR_DEPENDENCY_TREE = frozenset({<class 'llama.files.i3.json.IceCubeNeutrinoList'>})
property log_message

GraceDB upload log message for <class ‘llama.files.i3.json.IceCubeNeutrinoList’>

class llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList

Bases: llama.files.slack.SlackReceiptLlama

digraph "llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList" { charset="utf-8" splines=ortho bgcolor=none margin=0 rankdir=LR "llama.files.i3.json.IceCubeNeutrinoList" [label=<{{<B>IceCubeNeutrinoList</B>|<I>icecube_neutrino_list.json</I>|JSONFile<BR/>HEALPixPSF}}>, shape="record", style=filled, target=_top, URL="/llama.files.i3.json.html#llama.files.i3.json.IceCubeNeutrinoList", fillcolor="#cccccc"]; "llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList" [label=<{{<B>RctSlkLmaIceCubeNeutrinoList</B>|<I>rct_slk_lma_icecube_neutrino_list.json.json</I>|SlackReceiptLlama}}>, shape="record", style=filled, target=_top, URL="/llama.files.i3.json.html#llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList", fillcolor="#cccccc"]; "llama.files.skymap_info.SkymapInfo" [label=<{{<B>SkymapInfo</B>|<I>skymap_info.json</I>|EventTriggeredFileHandler<BR/>JSONFile}}>, shape="record", style=filled, target=_top, URL="/llama.files.skymap_info.html#llama.files.skymap_info.SkymapInfo", fillcolor="#cccccc"]; "llama.files.i3.json.IceCubeNeutrinoList" -> "llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList" [arrowtail="none", color="red", ]; "llama.files.skymap_info.SkymapInfo" -> "llama.files.i3.json.IceCubeNeutrinoList" [arrowtail="none", color="red", ]; "llama.files.skymap_info.SkymapInfo" -> "llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList" [arrowtail="none", color="red", ]; }

Required input files for llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList (located on the far right of the graph). For a typical trigger, the leftmost files will be generated as triggers become available. They will be used as input for files to their right, eventually allowing llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList to be generated.

DEPENDENCIES = (<class 'llama.files.i3.json.IceCubeNeutrinoList'>, <class 'llama.files.skymap_info.SkymapInfo'>)
FILENAME = 'rct_slk_lma_icecube_neutrino_list.json.json'
FILENAME_FMT = 'rct_slk_lma_{}.json'
MANIFEST_TYPES = (<class 'llama.files.i3.json.RctSlkLmaIceCubeNeutrinoList'>,)
UPLOAD

alias of IceCubeNeutrinoList

UR_DEPENDENCIES = (<class 'llama.files.skymap_info.SkymapInfo'>, <class 'llama.files.i3.json.IceCubeNeutrinoList'>)
UR_DEPENDENCY_TREE = frozenset({<class 'llama.files.i3.json.IceCubeNeutrinoList'>})
class_vetoes = ()
llama.files.i3.json.blinder(neutrino, unblinded_times=())

Randomize neutrino azimuth (and hence right ascension) for neutrinos triggered at times that are not explicitly approved for unblinded analysis.

Parameters

  • neutrino (dict) – A neutrino dictionary in the format returned by format_neutrinos.

  • unblinded_times (array_like, optional) – An iterable of [start, end] times (specified in UTC MJD format) in which we are approved by IceCube to pull unblinded neutrinos for our analysis. If not provided, it is assumed that ALL neutrinos should be blinded (DEFAULT: tuple()).

Returns

blinded – The same neutrino provided as a parameter with the azimuth (and hence right ascension) with RA scrambled if necessary. If the neutrino’s MJD falls inside one of these intervals, it will be returned unchanged. If it falls outside all of these intervals, its azimuth will be set to a random value. This is a fresh copy of the neutrino; the neutrino provided as input is not modified in place.

Return type

dict

Examples

>>> neutrino_time = 57990
>>> unblinded_times = [[neutrino_time-1, neutrino_time+1]]
>>> neutrino = {
...     "mjd": neutrino_time,
...     "zenith": 90,
...     "sigma": 0.3,
...     "energy": 5000,
...     "azimuth": 270,
...     "bdt_up": 1,
... }
>>> blind = blinder(neutrino, unblinded_times=[])
>>> blind['azimuth'] == neutrino['azimuth']
False
>>> unblind = blinder(neutrino, unblinded_times=unblinded_times)
>>> unblind['azimuth'] == neutrino['azimuth']
True
llama.files.i3.json.format_neutrinos(neutrinos)

Read in ICECUBE neutrinos as returned by realtime_tools.live.get_events and put them in the format that LLAMA expects.

llama.files.i3.json.get_archive(mjd_start, mjd_end)

Fetch neutrinos from a local archive. Archive must be defined and stored on LLAMA S3 for the queried neutrinos, and you must have access to LLAMA S3 API credentials to retrieve the archive.

Parameters

  • mjd_start (float) – The start of the time window in which to search for neutrinos in Modified Julian Date (MJD) format. Use gps2mjd or utc2mjd from the llama.utils module to make the time conversions to the appropriate format.

  • mjd_end (float) – The end of the time window in which to search for neutrinos.

Returns

formatted – A list of neutrino dictionaries formatted as expected by IceCubeNeutrinoList. If mjd_start > mjd_end, no error will be raised; instead, an empty list of neutrinos will be returned.

Return type

list

Raises

FileNotFoundError – If data for the requested time range is not available.

Examples

You can confirm that the archival neutrino data is available by fetching a specific trigger from the archive: >>> neutrino = get_archive(58392, 58393)[0] >>> neutrino[‘run’] 131569 >>> neutrino[‘event’] 43188689

llama.files.i3.json.get_archive_and_real_neutrinos(mjd_start, mjd_end, blinded_neutrinos=None, check_complete=False)

Get a mix of archival and real online GFU neutrinos (automatically calls get_archive and get_real as appropriate). Pulls online neutrinos for dates after MJD_TOPIC_SWITCH_17_TO_BLANK and archival neutrinos for dates prior.

Parameters

  • mjd_start (float) – The start of the time window in which to search for neutrinos in Modified Julian Date (MJD) format. Use gps2mjd or utc2mjd from the llama.utils module to make the time conversions to the appropriate format.

  • mjd_end (float) – The end of the time window in which to search for neutrinos.

  • blinded_neutrinos (bool, optional) – Whether the neutrinos should be forcefully blinded or not. Use this to blind the neutrinos even if they are from time windows approved for unblinding (e.g. when doing background sensitivity studies). If not specified, use UNBLINDED_TIMES to determine which neutrinos can be unblinded.

  • check_complete (bool, optional) – Neutrinos from IceCube can be delayed due to communications issues; to combat this, when neutrinos are pulled from the online data stream and check_complete is True, a check will be run to see if the next neutrino after the end of the search window is yet available. If it is not available, it is assumed that not all neutrinos needed for the search have arrived from south pole, and a GenerationError will be raised, allowing the pipeline to cooldown and retry the neutrino pull later.

Returns

formatted – A list of neutrino dictionaries formatted as expected by IceCubeNeutrinoList. If mjd_start > mjd_end, no error will be raised; instead, an empty list of neutrinos will be returned. Each neutrino will be blinded if blinded_neutrinos is True or if its event time falls outside the list of approved UNBLINDED_TIMES.

Return type

list

Examples

Pull some blinded archival neutrinos >>> a_start = MJD_TOPIC_SWITCH_17_TO_BLANK-1.77 >>> a_end = MJD_TOPIC_SWITCH_17_TO_BLANK >>> arc = get_archive_and_real_neutrinos(a_start, a_end, True) >>> len(arc) > 0 True

Pull some blinded GFU neutrinos >>> l_start = MJD_TOPIC_SWITCH_17_TO_BLANK >>> l_end = MJD_TOPIC_SWITCH_17_TO_BLANK + 500/86400. >>> live = get_archive_and_real_neutrinos(l_start, l_end, True) >>> len(live) > 0 True

Pull a mix of archival and blinded neutrinos >>> mix = get_archive_and_real_neutrinos(a_start, l_end, True) >>> len(arc) + len(live) == len(mix) True

llama.files.i3.json.get_dec_filter(dec)

Return a function that will filter neutrinos based on DECLINATION, e.g. for use with the filter function.

Parameters

dec (list) – A list of declination intervals (also specified as lists) into which a selected neutrino should fall, e.g. [[ 21, 41 ],[ 92, 102 ]].

Returns

dec_filter – A function which returns True when passed a neutrino that falls within at least one of the declination intervals passed included in dec. The neutrino is expected to be in the format returned by format_neutrinos.

Return type

function

Examples

>>> neutrino = {
...     'azimuth': 246.16444118441663,
...     'bdt_up': -0.0155,
...     'energy': 498.10789,
...     'mjd': 57982.5321185039,
...     'sigma': 1.5478695443038595,
...     'zenith': 113.7504569829126
... }
>>> ra, dec = zen_az2ra_dec(
...     neutrino['mjd'],
...     neutrino['zenith'],
...     neutrino['azimuth']
... )
>>> f"RA: {ra:.3f}, Dec: {dec:.3f}"
'RA: 1.253, Dec: 23.653'
>>> get_dec_filter([[0,30],[50,60]])(neutrino)
True
>>> get_dec_filter([[0,20],[50,60]])(neutrino)
False
llama.files.i3.json.get_ra_filter(ra)

Return a function that will filter neutrinos based on RIGHT ASCENSION, e.g. for use with the filter function.

Parameters

ra (list) – A list of right ascension intervals (also specified as lists) into which a selected neutrino should fall, e.g. [[ 21, 41 ],[ 92, 102 ]].

Returns

ra_filter – A function which returns True when passed a neutrino that falls within at least one of the right ascension intervals passed included in ra. The neutrino is expected to be in the format returned by format_neutrinos.

Return type

function

Examples

>>> neutrino = {
...     'azimuth': 246.16444118441663,
...     'bdt_up': -0.0155,
...     'energy': 498.10789,
...     'mjd': 57982.5321185039,
...     'sigma': 1.5478695443038595,
...     'zenith': 113.7504569829126
... }
>>> ra, dec = zen_az2ra_dec(
...     neutrino['mjd'],
...     neutrino['zenith'],
...     neutrino['azimuth']
... )
>>> f"RA: {ra:.3f}, Dec: {dec:.3f}"
'RA: 1.253, Dec: 23.653'
>>> get_ra_filter([[0, 30], [50, 60]])(neutrino)
True
>>> get_ra_filter([[5, 30], [50, 60]])(neutrino)
False
llama.files.i3.json.get_real(mjd_start, mjd_end, check_complete=False)

Fetch real neutrinos from ICECUBE’s database; takes gps time of the events and the time range abnut this central value over which to search for neutrinos.

Parameters

  • mjd_start (float) – The start of the time window in which to search for neutrinos in Modified Julian Date (MJD) format. Use gps2mjd or utc2mjd from the llama.utils module to make the time conversions to the appropriate format.

  • mjd_end (float) – The end of the time window in which to search for neutrinos.

  • check_complete (bool, optional) – Neutrinos from IceCube can be delayed due to communications issues; to combat this, when check_complete is True, a check will be run to see if the next neutrino after the end of the search window is yet available. If it is not available, it is assumed that not all neutrinos needed for the search have arrived from south pole, and a GenerationError will be raised, allowing the pipeline to cooldown and retry the neutrino pull later.

Returns

formatted – A list of neutrino dictionaries formatted as expected by IceCubeNeutrinoList. If mjd_start > mjd_end, no error will be raised; instead, an empty list of neutrinos will be returned.

Return type

list

Raises

ValueError – If the start time of the neutrino pull predates the switch from the ‘neutrino17’ topic to the ‘neutrino’ topic (in late 2018), a ValueError is raised. The date of this switch is given by MJD_TOPIC_SWITCH_17_TO_BLANK.

Examples

Get some neutrinos from late-2018 on the ‘neutrino’ topic: >>> from llama.utils import utc2mjd >>> mjd = utc2mjd(“2018-09-30”) >>> len(get_real(mjd-500/86400., mjd+500/86400.)) > 0 True

If your end time is earlier than your start time, you’ll get an empty neutrino list: >>> get_real(mjd, mjd-1) []

You should be able to fetch this neutrino from the online GFU database: >>> neutrino = get_archive(58392, 58393)[0] >>> neutrino[‘run’] 131569 >>> neutrino[‘event’] 43188689

llama.files.i3.json.neutrino_archive_available_years()

Get a list of years for which archival GFU neutrinos are available. These files are stored remotely on LLAMA S3 (see llama.com.s3) as .npy files and are cached locally in llama.utils.OBJECT_DIR; these are private files that require LLAMA S3 credentials to access. Archival neutrino releases available via LLAMA S3 are recorded in ARCHIVAL_NEUTRINOS, and the default archival release version is contained in DEFAULT_ARCHIVE.

Examples

Confirm that we have access to archival neutrinos from 2011 through 2018:

>>> neutrino_archive_available_years()
[2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018]
llama.files.i3.json.random() → x in the interval [0, 1).