Developer Installation (Pre-O3)

NOTE: Theses instructions are for older versions of the pipeline (pre-O3). They are kept in place in case you need to work with those versions, or in case you need a starting point for e.g. a full IceCube stack installation. If you are working with the latest versions of the code, please read previous sections of the developer guide.

System Requirements

Make sure you have at least 4GB of memory (physical or virtual; `swap space`_ is fine) and 15GB of free space on your file system.

Introduction

To get up and running, start up a bash session and run each of the commands below. You should log in as a user other than root (more precisely, you should log in as the user who will run the LLAMA software in production; on a new install, you might need to create a new user). If you are installing LLAMA on a remote server (i.e. a computer besides the one that you are currently sitting in front of), you can read the SSH with X11 Forwarding section for instructions on how to connect to a remote server via SSH.

The below instructions will only work on Debian 8 (Jessie). You can adapt them to other platforms if you know what you are doing, but Debian 8 is the only supported platform. It should be possible to finish installation by blindly running commands, but even so, make sure to copy and paste each line one-at-a-time to make sure they are entered properly. Don’t worry about the explanatory bits if you are not interested, but make sure to read parts that are in bold. You don’t have to click any of the hyperlinks in this guide; they are for reference purposes only. If you encounter trouble, look in the Appendix.

Installing LLAMA Dependencies

There are a few dependencies you will need for LLAMA installation and development. Install these now:

sudo apt-get -y update
sudo apt-get -y install python-sphinx rsync curl wget unzip git dtrx \
    msmtp-mta heirloom-mailx xpdf debconf-utils make \
    apache2 apache2-doc apache2-utils ncdu ack-grep silversearcher-ag \
    python3-six python3-pytest \
    python3-pytest-cov ipython3 pandoc libapache2-mod-python \
    latexmk python-profiler python-wxgtk3.0 python-setuptools runsnakerun \
    htop texlive-publishers

You will also want to install git-lfs, an extension to git used for large file storage (git-lfs install instructions taken from here), which LLAMA uses to store large data files:

curl -s \
    https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh \
    | sudo bash
sudo apt-get -y install git-lfs

You must activate git-lfs on a per-user basis. Make sure to run the following command as the user who will run the LLAMA server:

cd ~
git lfs install

Obtaining the LLAMA Software (pre-O3)

LLAMA software is stored in a git repository with git-lfs used for large data file storage. You can obtain the entire LLAMA software repository with a quick git clone followed by a somewhat slower fetching of the large data files. First, though, you will want to Generate SSH Keys so that you don’t need to enter your password over and over (SSH keys are actually required on git.ligo.org and, at time of writing, on the Bitbucket multimessenger pipeline repository). See the Bitbucket Authentication and git.ligo.org Authentication sections for instructions on adding your SSH keys to those sites.

Once you’ve set up SSH keys, it’s time to actually clone the LLAMA repository:

cd ~
git clone git@bitbucket.org:stefancountryman/multimessenger-pipeline.git

You might be prompted to confirm that you want to connec to Bitbucket; if you see something like the following, just type yes and hit enter:

The authenticity of host ‘bitbucket.org (18.205.93.0)’ can’t be established.

RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40.

Are you sure you want to continue connecting (yes/no)?

You can (pretty much) confirm that the LLAMA software folder was created by running:

cd multimessenger-pipeline && git status; cd ~

If you see the following response:

On branch master

Your branch is up-to-date with ‘origin/master’.

nothing to commit, working directory clean

then you have (most likely) succeeded in obtaining the LLAMA codebase.

Setting up Configuration Files

You must now move the default configuration files into place (this affects things like gw-astronomy.org access and email sending):

ipython profile create
ln -s \
    ~/multimessenger-pipeline/static/llama-ipython-startup.py \
    ~/.ipython/profile_default/startup/llama-ipython-startup.py
ln -s multimessenger-pipeline/static/.gitattributes ~
ln -s multimessenger-pipeline/static/.inputrc ~
ln -s multimessenger-pipeline/static/.mailrc ~
cp multimessenger-pipeline/static/.msmtprc ~
cp multimessenger-pipeline/static/.netrc ~
cp multimessenger-pipeline/static/.twilio-credentials ~
ln -s multimessenger-pipeline/static/.vimrc ~
mkdir -p ~/.ssh
ln -s multimessenger-pipeline/static/.ssh/config ~/.ssh
chmod 0600 ~/.msmtprc
touch ~/.bashrc
[ -e ~/.bashrc.orig ] || cp ~/.bashrc ~/.bashrc.orig
cat \
    ~/.bashrc.orig \
    ~/multimessenger-pipeline/static/.bashrc-llama-addendum \
    >~/.bashrc
. ~/.bashrc

You should see “LLAMA” appear in big letters accross your terminal. You should also now be able to access the main LLAMA executable, llama, which contains all user features as subcommands, as well as developer scripts, since they should all now be part of your shell’s ${PATH} variable.

Now, you will want to preseed your Kerberos and HTCondor preferences to avoid being bugged later by debconf during your eventual Kerberos installation:

sudo debconf-set-selections ~/multimessenger-pipeline/static/preseed.cfg

Install git Hooks

You can have the server automatically regenerate documentation and home page when they are modified by installing the included git hooks. From the repository directory, run:

for h in git-hooks/*; do ln -fs ../../$h .git/hooks; done

Install LLAMA dependencies

You can install (mostly non-LIGO) LLAMA dependencies from the requirements.txt file:

curl -O https://raw.githubusercontent.com/stefco/llama-env/master/requirements.txt
pip install -r requirements.txt
rm requirements.txt

Install LIGO Software

LIGO supports Scientific Linux, Debian, and (on a best-effort basis) OS X. The below instructions are for Debian 8 (Jessie).

First, we will need to install LIGO dependencies. The first step is adding LIGO’s package repositories to Debian’s list of sources so that Debian’s package manager knows where to find the LIGO software packages we are about to install. This simply requires copying the source list into Debian’s source list directory. Further documentation available on LIGO’s Software Downloads page.

sudo cp ~/multimessenger-pipeline/static/lscsoft.list \
    /etc/apt/sources.list.d/lscsoft.list

The following line should prevent the installation process from asking for user feedback. This is useful if you want to run installation in the background.

export DEBIAN_FRONTEND=noninteractive

You should always update your package manager’s list of repositories before trying to install something new:

sudo apt-get -y -qq update

You will need to tell the package manager to trust LIGO software. You might still get warnings about untrusted software packages; this is fine.

sudo apt-get install -y -qq --force-yes lscsoft-archive-keyring

Update again.

sudo apt-get -y -qq update

Finally, install lscsoft-all, the comprehensive LIGO software package.

sudo apt-get install -y -qq --force-yes \
    -o Dpkg::Options::="--force-confdef" \
    -o Dpkg::Options::="--force-confold" \
    lscsoft-all lalinference

Next, install ligo datagrid; this usually fails the first time and works the second time for some reason. If it fails, the below command will keep retrying until it succeeds, which generally seems to work.

retval=1
until [ $retval -eq 0 ]; do
    echo 'ATTEMPTING TO INSTALL DATAGRID'
    curl https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/ldg-client.sh \
        >/tmp/ldg-client.sh \
        && sudo bash /tmp/ldg-client.sh
    retval=$?
done

Once the script is finished executing, you can confirm that installation was successful by running:

type ligo-proxy-init >/dev/null 2>&1 \
    && { echo "LIGO Data Grid Installation succeeded."; } \
    || { echo "LIGO Data Grid Installation failed! Try again."; }

to check for one of the installed executables. This command will tell you whether installation was successful (again, if installation failed, just try repeating the previous step).

Finally, install miscellaneous python dependencies using pip. One of these dependencies, gwpy, is in active development and can be a bit finicky. The below instructions should “just work”, but if not, further documentation is available on GWPy’s install instructions page.

Tell Debian to use the default locale (otherwise errors come up in the scipy update):

export LC_ALL=C

Update the python package installer pip:

sudo pip install --upgrade pip

Update scipy to a version recent enough for GWpy to work (the next line might take a while to finish and might produce a lot of ugly output depending on your precise version of Debian Jessie; don’t be alarmed if this is the case.):

sudo pip install 'scipy>=0.16'

Install GWpy:

sudo pip install gwpy

Install IceCube Offline Software

These packages are used for retrieving neutrino data from IceCube.

Installing IceCube Dependencies

First you’ll need to make sure you’re using a newer version of cmake than is available on Debian Jessie; do this by adding more recent backports to the apt sources list:

sudo cp ~/multimessenger-pipeline/static/backports.list \
    /etc/apt/sources.list.d/backports.list

Now you can install the latest cmake version by forcing apt-get to use the jessie-backports repository:

sudo apt-get -t jessie-backports install cmake

Now, install dependencies for the IceCube offline software (with all of the recommended extras):

apt-get install build-essential cmake libbz2-dev libgl1-mesa-dev \
    freeglut3-dev libxml2-dev subversion libboost-python-dev \
    libboost-system-dev libboost-signals-dev libboost-thread-dev \
    libboost-date-time-dev libboost-serialization-dev \
    libboost-filesystem-dev libboost-program-options-dev \
    libboost-regex-dev libboost-iostreams-dev libgsl0-dev libcdk5-dev \
    libarchive-dev python-scipy ipython-qtconsole libqt4-dev python-urwid \
    libz-dev libqt5opengl5-dev libstarlink-pal-dev python-sphinx \
    libopenblas-dev libcfitsio3-dev libsprng2-dev libmysqlclient-dev \
    libsuitesparse-dev libcfitsio3-dev libmysqlclient-dev \
    libhdf5-serial-dev root-system

Entering IceCube Credentials

Next, store your IceCube username and password in the uname and pword variables; we will use this to check out the IceCube repository. You should have received IceCube login credentials as part of the installation procedure. The next two lines of commands will read your IceCube username and password in and store them as variables. If you make a mistake entering your password, or if something goes wrong while installing IceCube software, you should run these lines again just to make sure that your credentials are being passed to SVN in the next step.

read -p 'Enter your IceCube username: ' uname && \
read -sp 'Enter your IceCube password: ' pword && echo

Next, check out IceCube offline software. These instructions are adapted from the original IceCube offline software installation instructions; you can find all offline software releases here. A full overview of IceCube software is available here. Make sure to check out the latest version (if it is not the same as the one listed in the following code block). The while loop is included to catch and handle segmentation faults during checkout; these are, unfortunately, a known bug in Debian Jesse’s SVN version.

sudo mkdir -p /usr/local/icecube/offline
cd /usr/local/icecube/offline
export IceCube_SVN=http://code.icecube.wisc.edu/svn
sudo svn --username "$uname" --password "$pword" co \
    "$IceCube_SVN"/meta-projects/offline-software/releases/V18-06-00 src

You should run the next code block to make sure that the SVN checkout succeeded in full.

retval=$?
while ! [ $retval -eq 0 ]; do
    echo 'QUIT EARLY DUE TO SEGFAULT IN SVN, RESTARTING CHECKOUT'
    sudo svn cleanup src
    sudo svn update src
    retval=$?
done

Now it is time to build the IceTray software. This next step is very slow; let this run for a couple of hours while you go do something else. Fortunately, this step can be resumed at the make command in the event of failure.

sudo mkdir -p /usr/local/icecube/offline/build
cd /usr/local/icecube/offline/build
sudo cmake -DSYSTEM_PACKAGES=True ../src
sudo make

If you get an error while running cmake or make, there was probably an error that prevented you from fetching the entire software repository from SVN. You should go back a few steps and start over from when you entered your IceCube credentials a few steps ago. Finally, download some data used by IceCube test and example scripts.

sudo make rsync

The .bashrc modification you made at the beginning of this whole procedure has already put the necessary modifications to your environmental variables in place (e.g. your PYTHONPATH variable has been updated to include the location of the new IceCube python libraries, so that you can import them next time you run python). You can therefore test whether the IceCube python software was installed successfully:

python -c 'from icecube import dataclasses; print("Success!")'

You should see Success! printed on your console. If so, congratulations! You have successfully installed the IceCube offline software. Now, you will have to check out a set of neutrino querying python tools used by LLAMA. Change to the LLAMA software directory and check out the live software. Again, store your IceCube username and password in variables (you can skip this step if those variables are still initialized from before):

read -p 'Enter your IceCube username: ' uname && \
read -sp 'Enter your IceCube password: ' pword && echo

Now, check out the live querying software:

cd ~/multimessenger-pipeline/icecube
export IceCube_SVN=http://code.icecube.wisc.edu/svn
svn --username "$uname" --password "$pword" co \
    "$IceCube_SVN"/projects/realtime_tools/releases/V19-02-00/python \
    realtime_tools
svn --username "$uname" --password "$pword" co \
    "$IceCube_SVN"/projects/realtime_gfu/releases/V19-02-00/python \
    realtime_gfu
pushd realtime_gfu
svn --username "$uname" --password "$pword" co \
    "$IceCube_SVN"/projects/realtime_gfu/releases/V19-02-00/resources \
    resources
popd
# this is a kludge to get the realtime_gfu resources in place
sudo cp -R \
    ~/multimessenger-pipeline/icecube/realtime_gfu \
    /usr/local/icecube/offline/build/

Install MATLAB

You will need to download the MATLAB Installer as well as the JSONLab toolbox. You must install the following three MATLAB Toolboxes at the same time that you install MATLAB, and you must make sure to also install MATLAB itself. Do so by selecting the following four items in the install dialog (note that the MATLAB version might change from 9.1, but it should be at the top of the list):

1. MATLAB 9.1

2. Image Processing Toolbox

3. Mapping Toolbox

4. Statistics and Machine Learning Toolbox

Also make sure to check the box in the install dialogue asking if you would like to make symlinks to MATLAB. This is necessary in order for command line calls to matlab to work (which is necessary for our scripts to run in the current iteration of the pipeline). If you forget to do this, you can manually add the symlink in with:

sudo ln -s /usr/local/MATLAB/\*/bin/matlab /usr/local/bin/matlab

You can confirm that the symlink was made by running:

type matlab 2>/dev/null 1>&2 && echo 'success!' || echo 'symlink not found!'

You can install the JSONLab toolbox from within the MATLAB interface after it has been successfully installed. See the appendix for some MATLAB installation details. You will usually have to install JSONLab twice before it remains permanently installed. It is not clear why this is, but it works after the second installation. You can confirm that JSONLab is installed by opening a MATLAB session and seeing if loadjson is a valid command; if it is, then JSONLab has been successfully installed. You can also simply run this command to see if JSONLab installed successfully:

matcmd=""
matcmd+="if exist('loadjson')"
matcmd+="    disp('JSONLab installed successfully.');"
matcmd+="else"
matcmd+="    disp('JSONLab failed to install.');"
matcmd+="end;"
matcmd+="exit"
matlab -nodesktop -nosplash -noFigureWindows -r "${matcmd}"

If JSONLab failed to install, just open up MATLAB and reinstall it. This is sometimes necessary for some reason.

Now, install the matlab python module, which will allow python code to call MATLAB scripts via the MATLAB Engine API:

cd $(sudo find / -path '\*/extern/engines/python')
sudo python setup.py install

Confirm that installation of the Python MATLAB Engine API succeeded:

python -c 'import matlab; print("success!")'

Configuration and Authentication

These steps involve entering your credentials for the services used by LLAMA, as well as some basic configuration. These instructions assume that you have LIGO login credentials. Some of the steps require you to contact one of our partners (e.g. GCN or gw-astronomy.org) to configure authentication. These appear after this section in the external authentication section.

Generate SSH Keys

SSH keys are used to securely log in to other computers without a password. LLAMA uses them to upload data to gw-astronomy.org. A great and user-friendly guide to setting up SSH keys is available on Digital Ocean’s blog. These instructions are pulled from that article.

First, check whether you have an SSH key already created. The following command will print “no SSH key found” if it doesn’t find an existing key:

[ -e ~/.ssh/id_rsa.pub ] || echo "no SSH key found."

If you don’t already have an SSH key, create a new one by running the following command and hitting enter repeatedly until it finishes (WARNING: this will overwrite your existing SSH key if you do in fact have one already, possibly costing you access to servers you are currently able to SSH into.):

ssh-keygen

You should see some funny ASCII art print out once it’s finished. You can now print your SSH key with:

cat ~/.ssh/id_rsa.pub

You will eventually have to make sure that the public key is on gw-astronomy.org’s list of authorized keys; this step is discussed below in the external authentication section.

GMail Authentication

1. Create a device (or “app”, as Google calls it) password for gw.hen.analysis@gmail.com (or your personal gmail, if you are just trying to run some tests). This password is different from your login password and is only to be used by a single device. You can generate a device password here.

2. Use your favorite text editor to open ~/.msmtprc and replace <WRITE-PASSWORD-HERE> with your newly-created device password. If you are using a personal gmail account (rather than gw.hen.analysis@gmail.com), you will want to find each occurence of gw.hen.analysis@gmail.com in this file and replace it with your personal gmail address.

LIGO Authentication

1. Run ligo-proxy-init your.username and enter your LIGO password. This will allow you to download GW event data from GraceDB. You will need to do this every few days.

LV_Alert Authentication

1. Make sure you have an LVAlert username and password set up. You can do this here. You can learn more about LVAlert on the official documentation pages and from the GraceDB LVAlert guide.

2. Use your favorite text editor to open ~/.netrc and replace <WRITE-USERNAME-HERE> with the LVAlert username you created in step 1 and <WRITE-PASSWORD-HERE> with the password for that LVAlert username. If you already have a password but forgot it, you can easily reset it at the link provided in step 1.

3. (Optional) You can make sure that your credentials work by running

 lvalert_admin --username albert.einstein --subscriptions

where, of course, ``albert.einstein`` is replaced with the LVAlert username
you made in step 1.

Setting Up SSL Certificates

In order to access the server using the HTTPS protocol (as is generally considered best practice, since this will traffic to/from the server when accessing it via browser), you’ll need to get an SSL certificate. The easiest way to do this is by using the EFF Certbot (linked instructions for Debian Jessie, but just use the correct ones for whatever system is running the server) and using Let’s Encrypt.

These instructions will let you install certbot-auto from EFF’s website. certbot-auto can automatically install certificates for an Apache server:

sudo /usr/local/bin/certbot-auto --apache

You can then automate renewal of the certificate (important because the certs expire after 90 days) by adding the following to your systemd timer or root crontab:

/usr/local/bin/certbot-auto renew

Setting Up Passwords for Run Summary Pages

You can put the username and password for the current run in ~/.llama_run_credentials in .netrc format. This will be used for the run summary pages when basic HTTP authentication is in use with the flask dev server.

External Authentication

The services referenced in this section require credentials provided by LLAMA partners and commercial services. Authenticating with these services is not necessary for testing the majority of the LLAMA pipeline’s functionality, but it does require either expenditure of funding (for Twilio) or getting help from our busy partners (for gw-astronomy.org and GCN). These are live services that are used to disseminate results, and testing their functionality requires care and coordination. It is therefore highly recommended that this section be skipped by all users who are not actively deploying the LLAMA code into production.

Bitbucket Authentication

At time of writing, the main software repository is stored on Bitbucket.

The first step is to Generate SSH Keys; once you’ve done that, you can access your SSH key by logging in to your LLAMA server and printing out your public key:

cat ~/.ssh/id_rsa.pub

You can now copy this key straight from your terminal and add it to Bitbucket. First, navigate to Bitbucket’s website, followed by your user settings page, and then click the SSH keys section under Security. At time of writing, you can also directly access these settings at the following URL, where USERNAME has been replaced with your bitbucket username:

https://bitbucket.org/account/user/USERNAME/ssh-keys/

Once there, click Add key and type the name of your server under Label and paste the SSH key into the Key section. Hit Add key to finish the process. You should now be able to push and pull from your Bitbucket repositories at the command line on your server without needing to enter your username and password, and if you have permission to access the pipeline repository, you will now be able to clone it from its Bitbucket URL.

git.ligo.org Authentication

The first step is to Generate SSH Keys; once you’ve done that, you can access your SSH key by logging in to your LLAMA server and printing out your public key:

cat ~/.ssh/id_rsa.pub

You can now copy this key straight from your terminal and add it to git.ligo.org. Visit your git.ligo.org SSH-key settings page. Once there, type the name of your server under Title and paste the SSH key into the Key section. Hit Add key to finish the process. You should now be able to push and pull from your git.ligo.org repositories at the command line on your server without needing to enter your username and password, and if you have permission to access the pipeline repository, you will now be able to clone it from its Bitbucket URL.

Twilio Authentication

We use Twilio’s API to send SMS messages to LLAMA operators for human-in-the-loop parts of the pipeline. To authenticate with Twilio, you will need to provide your credentials as environmental variables. The .bashrc file provided by default will try to run ~/.twilio-credentials, so the easy way to do this is to put your credentials in that file. That file is pre-populated with the current LLAMA Twilio phone number, so you don’t have to provide that.

1. Go to Twilio’s dashboard and click “Show API Credentials” in the top right corner of the screen. Take note of the Account SID and Auth Token.

2. Open up ~/.twilio-credentials with your prefered text editor

3. Replace <YOUR-SID-HERE> with the Account SID you just got from Twilio’s website. Replace <YOUR-TOKEN-HERE> with the Auth Token you just got and save the file.

4. You will need to reload your credentials or your ~/.bashrc file for the changes to take effect. You can reload just the credentials by running:

source ~/.twilio-credentials

GW Astronomy Authentication

Make sure that the llama user on the gw-astronomy.org server has your public key saved to its authorized_keys file. If you have not set this up yet, you will need to email the webmaster at gw-astronomy.org and send your public key, ~/.ssh/id_rsa.pub, so that the webmaster can authorize you to log in to LLAMA’s gw-astronomy.org account. This is necessary for uploading analysis results. If you don’t know how to generate SSH keys, refer to the instructions showing how to Generate SSH Keys above.

GCN Authentication

Make sure that you have an entry for this server in the GCN network’s Sites Configuration File. This is necessary for receiving LVC events. In particular, you need to make sure there is a site entry containing your server’s IP address (which must be static) under the DIST_ADDRESS as well as a properly set LVC_Enabled field. You should also make sure you are set up to receive the proper alerts (including LVC_INITIAL_POS_MAP and GWHEN_COINC at the time of writing).

Turning on the Pipeline

Subscribing to LVAlert Nodes

Make sure you have subscribed to all of the LVAlert nodes used by the pipeline by running:

lvalert_admin -i

You should see all of the following (ordering does not matter):

Node: superevent [ subscribed subid=k7hjUhAX0qLiLjXL6bCnXDPT1JngsDwFmZvCdGqp]
Node: cbc_gstlal [ subscribed subid=DRPc5g1ivFNn504a5uQk2mzHVdarHWVzxD6S3u8Y]
Node: cbc_lowmass [ subscribed subid=DGYiTkwsl58w3fwsFX72g2pIqERg0jfEY7fW9Bop]
Node: stc-testnode [ subscribed subid=7lzckGSYHH28wswE4Rl8JnlpncKi9SsL8uOJZzZa]
Node: cbc_pycbc [ subscribed subid=CDQriRVw7IsYU6Uh2Y738oHlwPnXkJJvdD9PWeTQ]
Node: cbc_mbtaonline [ subscribed subid=cfiMONixdKRnl2DmSupPSD0BSArF3PasuoMAPcFU]
Node: burst_cwb [ subscribed subid=LpOfqpgwdnNNn5XnRW9jNPCn7UZuxmtYJW7jlS7Z]
Node: test_superevent [ subscribed subid=T8W2oLnNMC8TUnwaSdBErJEG0My8VGlJEcnGMWFv]
Node: cbc_spiir [ subscribed subid=0H4qi2Fh5Uo7OnLjPaT1aydQLKiNdj0gxzmv5TXB]

If any of the above nodes are missing, add them with llama_lvalert_subscribe, which just calls lvalert_admin --subscribe --node ... on every node we want to listen to. It’s really just a shortcut for the following:

lvalert_admin --subscribe --node superevent
lvalert_admin --subscribe --node cbc_gstlal
lvalert_admin --subscribe --node cbc_pycbc
lvalert_admin --subscribe --node cbc_mbtaonline
lvalert_admin --subscribe --node cbc_spiir
lvalert_admin --subscribe --node cbc_lowmass
lvalert_admin --subscribe --node stc-testnode
lvalert_admin --subscribe --node burst_cwb
lvalert_admin --subscribe --node test_superevent

(Note: you can also always print a full list of available LVAlert nodes by running lvalert_admin -m.)

Starting Pipeline Daemons

Once you’ve subscribed to all of the above LVAlert nodes, navigate to the repository directory and run:

llama run
llama listen lvalert
llama listen gcn
nohup serve_current_run 1>/dev/null 2>&1 </dev/null &

These commands will start:

1. The LVAlert listener, which listens for new LVC events and uses them to create new event directories when skymaps become available; mark events as EM_READY when they have been vetted as ready for EM follow-up; and marks them as having a superevent associated with them, indicating that the results of a specific event can be distributed to the public under the name of that superevent.

2. The GCN Notice listener, which listens for new LVC events both as confirmation that the event is public (and hence LLAMA can send out an alert) and as a fallback to the LVAlert listener (in case something non-standard happened in creating the superevent–as has happened in the past with unusual events–which was corrected when the public-facing GCN Notice was sent out).

3. The LLAMA pipeline daemon (llama run), which checks the default event directory for non-test events (hence the filter for GraceIDs starting with “G”). When the GCN listener receives a new LVC VOEvent via a GCN Notice, the LLAMA pipeline daemon will detect the new VOEvent and start running the LLAMA joint analysis.

4. The Current Run status page server (serve_current_run), which serves a summary webpage with the current status of the LLAMA server and the events processed as part of the current run over port 5000. Assuming that the server is associated with the domain name gwhen.com, one can view the current status of the server (including memory usage and whether daemon processes are running) by visiting gwhen.com:5000.

(The < /dev/null part is necessary due to some strange behavior in MATLAB when running in the background using nohup.)

Testing LVAlert

Note that you can run a quick test to see if you’re connected to LVAlert by starting up lvalert_listen (as described above) and submitting a test event in GraceDB’s LVAlert JSON format using:

lvalert_send -v --node stc-testnode \
    --file ~/multimessenger-pipeline/static/mock_event.json

You should see the contents of that mock event show up in the LVAlert handler’s logs, and if there was some intended side-effect of that LVAlert, you should also see some verbose logging and a new/modified event in the default event directory (~/.local/share/llama/current_run/). Note that, because the test event was sent on stc-testnode, the event will be flagged as a test event, and though the pipeline should run, no alerts will be sent to the public.

Viewing Active Processes

You can see if these processes are running with:

llama run -R                    # find ``llama run`` processes
llama listen lvalert -R         # show lvalert listener processes (if any)
llama listen gcn -R             # show GCN listener processes (if any)
ps -ax | grep serve_current_run # find serve_current_run process

Checking Pipeline Logs

You can follow the log output in real time with:

lvalertd -t
tail -f logs/llamad.log
tail -f logs/serve_current_run.log

In general, all LLAMA pipeline output will be saved to the logs/ directory. This is true regardless of where you place the repository directory.

Killing Pipeline Daemons

You can shut down the pipeline processes with:

llama run -k
llama listen lvalert -k
llama listen gcn -k
ps -ax | grep serve_current_run | sed /grep/d | awk '{print $1}' | xargs kill

Deprecated: MATLAB Logs

When running the pipeline with MATLAB code, all MATLAB-specific logging should be viewable by running:

tail -f logs/llama.matlab.log   # matlab process output goes here

Developing for LLAMA

This section is intended for new LLAMA developers.

Introduction to Development

Structure of the Pipeline

The LLAMA pipeline is designed to be flexible, robust, reproducible, parallel, and east to develop. This is accomplished by treating data processing as a bunch of methods for processing input data and producing output data independently of other parts of the analysis. LLAMA defines a python class called a FileHandler that specifies at minimum the following information about a data file:

1. The filename

2. The complete list of input files required to generate this file

3. The code used to actually generate this file from its inputs

Thus, each file is connected to the list of files necessary for its own generation, forming a file dependency graph (an instance of a Directed Acyclic Graph, or DAG, as used in other data analysis pipeline tools). When a file’s dependencies are available, it will be generated without affecting neighboring parts of the analysis, and each step of the pipeline is reproducible using the saved files (which are the only stateful part of the pipeline).

Adding Functionality

This approach makes it very easy to add new data processing capabilities to LLAMA. Just follow these steps:

1. Add a new python file in /llama/files that defines a subclass of abstract.FileHandler You will have to implement a few abstract methods and properties (see e.g. llama/files/lvc_skymap_mat.py for a simple example of a FileHandler implementation):

   • The filename property, which gives the name of the new output file

   • The dependencies property, which returns a list of FileHandler classes for each input file required to generate this file (make sure to import those FileHandler classes!)

   • The _generate method, which defines the steps needed to generate this file using input data. You can access things like the full path to a dependency file by instantiating a copy of its FileHandler and getting the fullpath property using e.g. other_file_handler_class(self).fullpath.

2. Make sure that your new FileHandler is included in the actual pipeline by importing the class in /llama/files/__init__.py

3. Test that the pipeline runs and produces desired output by running one of the developer tests, e.g. make testpipeline7 (or whatever test you feel like using in the makefile). Check that the output data is what you would expect for your new file by looking in the event directory for this test (/testout/events/testpipeline7 in this example).

4. If your new file looks good, run nosetests in the root repository directory to make sure that you didn’t somehow break anything.

5. If nosetests pass, git commit your code and push it to the remote repository.

6. Last but not least, remember to restart the pipeline in order to see your updates actually applied; the pipeline does not refresh its code automatically (a good thing for developing). You can do so with the following commands:

llama run -k  # might need to run this twice
llama run

Developer Conventions and Best Practices

Please follow the following standards with code written for the LLAMA pipeline:

Data Format Conventions

• All event files must represent angles in degrees. This means that, when querying external APIs, you must convert radians to degrees before saving data in an event directory. (You should log any raw data you throw out so that we can check the correctness of the conversion; see point on logging below).

Coding Best Practices

• Don’t hard code filenames or other magic strings! If you need to get data from a file, make sure that that file has its own FileHandler defining how it can be (reproducibly!) generated, then import that FileHandler class, instantiate a copy by feeding the current FileHandler as an argument to the dependency FileHandler class, and get the data you want (like filename) from that instance.

• Log all actions. This can be done very easily by sticking to the FileHandler subclass idiom defined by LLAMA.

• Please verbosely log any incoming raw data before it is thrown out. This way we can go back and see if we accidentally deleted good pulled data.

Testing

Go into the root repository directory (i.e. the top level directory of the git repository, which is assumed to be ~/multimessenger-pipeline at the time of writing of this documentation) and run:

make testgracedb

to make sure that you are properly authenticated using your LIGO credentials. You should see the following output (or something like it) printed to console (note that the following is output; don’t type it into the console):

llama/tests/testgracedb.py | tee -a logs/llama.testgracedb.log
running testgracedb at 2016-09-13 18:32:05.577660
Attempting to connect to GraceDB.
Attempting to download the boxin day even event.log.
Received successfully. Logging contents.
Pipeline: gstlal
Search: HighMass
MChirp: 9.555
MTot: 26.3501410484
End Time: 1135136350.647757924
SNR: 11.710
IFOs: H1,L1
FAR: 3.333e-11

If your output does not match the above, then you most likely need to log in using your LIGO credentials (See LIGO Authentication).

Next, run the full collection of unit tests using:

nosetests

This will take a few minutes and will not produce much output while it is running. If all tests succeed, you will see the number of tests that were run and the status message OK. You will be alerted at the end if any of the tests fail or if the tests themselves cannot be completed due to errors. Be aware that, if any of the tests fails, the output files will remain in /tmp taking up a fair amount of space. They are hard links by default, so this shouldn’t take up much extra space, but it is worth deleting directories of the form /tmp/tmp* after a failed run (after making sure you don’t have other data matching that file pattern, of course).

Appendix

Migrating to Conda

Migrating to Conda should simplify some LIGO dependency issues. It might make it harder to use IceCube software (have not checked whether this supports Conda packaging yet), but this is not currently an issue since we have IceCube stub methods installed.

Note that, at time of writing, LVAlert had not migrated to Conda, and so the old installation method is still required to be able to use LVAlert.

LIGO Wiki Documentation

The latest documentation on installing from source should always be available from the LSCSoft Conda documentation page. The instructions below are pulled directly therefrom. We can also follow LSC’s guide for making a new Conda package if we want to make the pipeline distributable by Conda in the future.

Installing LIGO Software via Conda

Conda installs are done on a per-user basis, so you won’t need to use sudo for any of the below. Start by installing the latest version of Conda:

curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

You’ll now need to exit your SSH session and log in again for conda to show up as a command; run exit to exit and then SSH back into the server. Now, activate your Conda environment and add the conda-forge channel (used to distribute custom Conda packages):

conda activate
conda config --add channels conda-forge

Next, install all available LIGO software:

wget -q https://git.ligo.org/lscsoft/conda/raw/master/environment-py36.yml
conda env create -f environment-py36.yml

Finally, you can activate the LIGO python computing environment with:

conda activate ligo-py36

You will probably want to put this in your .bashrc if you plan on using it all the time.

Troubleshooting Installation

Problem: cmake keeps failing while installing IceCube software.

Solution: You probably did not check out the entire IceCube repository when you ran svn co. You will need to remove the IceCube source and build directories by running sudo rm -rf /usr/local/icecube/offline/* and start over from when you entered your IceCube credentials.

Creating a new user

You should preferably not run the pipeline as root. You can add a new user named, e.g., vagrant in Debian and Ubuntu with:

adduser vagrant

You will then want to give this user sudo priviledges by adding them to the sudoers file (this will allow the user to do administrative tasks using the command sudo). You will want to run

visudo

which will open up the sudoers file for editing in the default text editor (for Debian, this is probably nano). You will need to add the following line anywhere in the file, then save and quit:

vagrant ALL=(ALL) NOPASSWD: ALL

(Of course, if your username is not vagrant, you should use a different name above.)

You can test whether your update to sudoers succeeded by running:

sudo echo 'Success! You have sudo priviledges.'

If this succeeds, it will print a success message.

Out of Memory

Add swap space (virtual memory) using the following commands:

sudo fallocate -l 12G /swapfile
sudo chmod 0600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon -s
free -m
sudo bash -c \
    "echo '/swapfile    none    swap    sw    0    0' >>/etc/fstab"
cat /etc/fstab

MATLAB Installation Troubleshooting

You will probably want to install MATLAB using the GUI. This is easy if you are already on a desktop environment.

If you are configuring a remote server to run the LLAMA pipeline, you will need to make sure that you have X11 Forwarding enabled so that you can control the MATLAB installer GUI from your computer. See the appendix entry on SSH with X11 Forwarding for information.

Unzip the downloaded zip file with (note: the filename might change if you install a newer version of MATLAB.):

unzip matlab_R2016b_ginxa64.zip -d ~/matlab

and then run the install executable located in the unzipped directory, which will launch the MATLAB installer GUI:

~/matlab/install

Make sure to select items 1-3 (listed in the install MATLAB section) in the installer window in addition to MATLAB itself. You can do without all of the other toolboxes. Once you are done, run matlab with

matlab

which will bring up the MATLAB GUI. Download jsonlab, then change your MATLAB working directory to the folder containing jsonlab and double click on the jsonlab file, which should have a full name along the lines of jsonlab-1.2.mltbx. You will be asked if you would like to install jsonlab; you should, of course, say yes. Again, bear in mind that you might need to repeat this procedure; for some reason MATLAB doesn’t always “remember” that it has jsonlab installed when you restart it. I don’t know how to remedy this besides installing jsonlab for a second time. You can confirm that JSONLab is installed by opening a MATLAB session and seeing if loadjson is a valid command; if it is, then JSONLab has been successfully installed.

Install IceCube Offline Software with Root

At the original time of writing, it was not necessary to install IceCube offline software. The following instructions are necessary in the event that IceCube software libraries with ROOT dependencies become needed by LLAMA. For some reason, IceCube offline software will not compile properly when using the version of ROOT installed by the system package manager, so it is necessary to install the IceCube I3_PORTS version, as detailed bellow. This installation is extremely time consuming and should be skipped unless IceCube packages containing root become necessary for LLAMA.

First, install dependencies for the IceCube offline software (plus a few other requirements for LLAMA):

sudo apt-get -y install build-essential cmake libbz2-dev libgl1-mesa-dev \
    freeglut3-dev libxml2-dev subversion libboost-python-dev \
    libboost-system-dev libboost-signals-dev libboost-thread-dev \
    libboost-date-time-dev libboost-serialization-dev \
    libboost-filesystem-dev libboost-program-options-dev \
    libboost-regex-dev libboost-iostreams-dev libgsl0-dev libcdk5-dev \
    libarchive-dev python-scipy ipython-qtconsole libqt4-dev \
    python-urwid python-tables libxft-dev

Next, install I3 Ports, a set of executables that provide a consistent environment for IceCube offline software. IceCube documentation suggests running this as a user other than root. This next step will take several hours to complete and must be restarted if it fails partway through. You might consider letting this run overnight or during some other unsupervised stretch of time. It will be significantly quicker on a fast system, but still painfully slow.

cd ~
svn co http://code.icecube.wisc.edu/icetray-dist/tools/DarwinPorts/trunk \
    port_source
cd port_source
./i3-install.sh "$I3_PORTS"

Next, check out IceCube offline software. if you encounter a segmentation fault during SVN checkout, see the next set of instructions below the next code block. These instructions are adapted from the original IceCube offline software installation instructions; you can find all offline software releases here.

mkdir ~/offline
cd ~/offline
svn co http://code.icecube.wisc.edu/svn/meta-projects/offline-software/releases/V15-08-00 src

If you encounter a segmentation fault (a known bug on Debian Jesse’s SVN version), you can run svn cleanup src followed by svn update src and repeat until the source code has been fully checked out.

Now it is time to build the IceTray software. This step is also very long (if not as long as the previous one); set aside a couple of hours for it. Fortunately, this step can be resumed at the make step in the event of failure.

mkdir ~/offline/build
cd ~/offline/build
"$I3_PORTS/bin/cmake" ../src
make

Finally, download some data used by IceCube test and example scripts.

make rsync

You should now be able to open up an icecube environment using:

./env-shell.sh

You should see a “Welcome to Ice Tray” message. You can check whether the IceCube python module has been installed by running:

python -c 'from icecube import dataclasses; print("Success!")'

If this runs without error and prints Success!, then you have successfully installed the IceCube python module. Well done! You can now exit the Ice Tray environment:

exit

To make sure that IceCube software is available when logging in, you will have to add some new environmental variables to your .bashrc file with the install locations. Since environmental variables depend on where you installed the IceCube offline software, it is vital that you base the environmental variables off of your current installation configuration. Making sure that you are still in the build directory, run the following commands:

eval "$(grep ROOTSYS= env-shell.sh)"
_I3_BUILD="$(pwd -P)"
pushd ../src
_I3_SRC="$(pwd -P)"
popd
pushd "$I3_PORTS"
_I3_PORTS="$(pwd -P)"
popd
_I3_TESTDATA="$_I3_PORTS/test-data"
rm -f ~/.i3env
printf 'ROOTSYS="%s"' "$ROOTSYS" >>~/.i3env
printf 'PYTHONPATH="%s"/lib:"%s"/lib:"$PYTHONPATH"\n' \
    "$_I3_BUILD" "$ROOTSYS" >>~/.i3env
printf 'I3_BUILD="%s"' "$_I3_BUILD" >>~/.i3env
printf 'I3_SRC="%s"' "$_I3_SRC" >>~/.i3env
printf 'I3_PORTS="%s"' "$_I3_PORTS" >>~/.i3env
printf 'I3_TESTDATA="%s"' "$_I3_TESTDATA" >>~/.i3env

You can now test whether your environment will run correctly by sourcing your .bashrc file:

. ~/.bashrc

You should see a message telling you that the IceCube software environment was successfully configured. Quickly test whether the IceCube python module is installed in the same way as before:

python -c 'from icecube import dataclasses; print("Success!")'

Once again, you should see Success! printed on your console. If so, congratulations! You have successfully installed the IceCube environment.

Installing Ubuntu for Windows

With Windows 10, it is now possible to install an Ubuntu userspace (with Bash included) on a Windows machine. This means that Ubuntu packages can be installed and Ubuntu binaries can be run directly on Windows (thanks to some sort of system call translation wizardry). This provides a handy and straightforward way of connecting to remote servers via SSH, even if you are on Windows, without having to install something like Putty. This has some other nice advantages, and, for the purpose of this guide, I will assume you have Bash installed on your local computer (even if you are using SSH to connect to a remote server for installing the LLAMA software).

You can follow the instructions in this guide to install Ubuntu for Windows.

Logging in to a Remote Server Using SSH

Start by opening up a bash instance.

• If you are on a Mac, you can do this by running Terminal.app, which should be in your /Applications folder, or by hitting Command-Space, typing “Terminal.app”, and hitting Enter.

• If you are on a PC running Windows 10, you can actually install the Ubuntu userspace (with Bash included). Once this is installed, you can just open the Bash executable from your programs folder in the Start menu, or just hit the Windows button, type bash, and hit Enter to launch a new Bash session.

• If you are running Linux and don’t know how to start a terminal session, may God help you.

Now, let’s say you are logging in to a server at IP address 1.2.3.4, your username on the remote server is vagrant, and you have some password. Just run the following and enter your password when prompted:

ssh vagrant@1.2.3.4

If you have your server mapped to a web URL, like example.com, you can use that instead of the IP address, as show below:

ssh vagrant@example.com

Congratulations! You should be logged in to your remote server. You can now proceed with installing the pipeline.

Getting LLAMA Software onto a Remote Server

Let’s assume someone sent you the LLAMA software in a compressed archive (e.g. a zipfile). You should have received a download link for the archive, in which case you can download the file directly from your LLAMA server. If you are running a remote machine, log in using SSH; if you are running LLAMA locally or in a virtual machine, simply open up a terminal session.

Let’s assume the URL for the LLAMA software archive is example.com/llama.zip. You can download the archive to your home directory as follows:

cd ~
wget example.com/llama.zip

Now, you can unzip the file:

unzip llama.zip

You should now have a directory called multimessenger-pipeline in your home folder.

SSH with X11 Forwarding

This will allow you to run MATLAB’s installer GUI and the MATLAB GUI itself, both of which are necessary for the installation process, on a remote server or on a local virtual machine being accessed through ssh.

• On Macs, you must install XQuartz. Then, when logging in to the LLAMA server, use the -o ForwardX11=yes and -o ForwardX11Trusted=yes flags to turn on X Window Forwarding. For example, if your server is c137.com and the username is rick.sanchez, you can log in with:

  ssh -o ForwardX11=yes -o ForwardX11Trusted=yes rick.sanchez@c137.com
  

• On Linux, assuming you have a GUI, you probably have an X11 Server installed already. If that is the case, you can just ssh with the -o ForwardX11=yes and -o ForwardX11Trusted=yes flags, just as in the above example for Macs.

• On Windows machines, you can use XMing as your X Windows server and Putty as your SSH client. If you have Windows 10, you can use Ubuntu for Windows (See instructions for Installing Ubuntu for Windows), and you can use bash instead of Putty. Once again, you can run

  ssh -o ForwardX11=yes -o ForwardX11Trusted=yes rick.sanchez@c137.com
  

If you use your ~/.ssh/config file for ssh settings (will work for Linux or Mac; I am not sure about Windows), you can also automatically turn on X11 forwarding for your LLAMA server by adding the lines

ForwardX11 yes
ForwardX11Trusted yes

to the site entry for your LLAMA server in your ~/.ssh/config file. For example, if you wanted to call the above server earth, you could add the following block to your ~/.ssh/config file:

Host earth
    HostName c137.com
    User rick.sanchez
    ForwardX11 yes
    ForwardX11Trusted yes

Which makes connecting to the server with all of your desired settings much quicker (and saves you from having to remember ssh command syntax or the server URL):

ssh earth

Once you have established an SSH connection with X11 forwarding enabled, you will need to make sure that the root account and the account you are logging into share the same .Xauthority file. To do so, run:

cd ~
if sudo [ -e /root/.Xauthority ]; then
    sudo mv /root/.Xauthority /root/.Xauthority.orig
fi
sudo cp .Xauthority /root/.Xauthority

Using LLAMA

The llama python module and its executables all have docstring documentation and are designed for interactive use. Read the help functions and play around with things in ipython to get a feel for how they work, and ask Stefan Countryman for help if you have any questions. I will put FAQs here if I find that any Qs have become F.

Documenting LLAMA

Overview

LLAMA uses reStructuredText documents to generate nicely-formatted webpages and PDF files (via Sphinx). There is even a Makefile included to facilitate publishing. The instructions below assume that you are starting from within the LLAMA installation folder (if you followed this installation guide, it will be in ~/multimessenger-pipeline).

Publishing to gwhen.com Website

This repository contains tools for making a nice, reviewer/developer friendly website on the production server (gwhen.com at time of writing). Check out the README.md file in ~/multimessenger-pipeline/review-site for instructions. Also make sure that the contents of multimessenger-pipeline/git-hooks has been copied to multimessenger-pipeline/.git/hooks (to ensure that documentation is automatically refreshed whenever the documentation source is changed).

You can also enable the status server at gwhen.com/status. First, turn on CGI scripts:

sudo a2enmod cgi

You’ll then need to add this alias to your apache configuration file (should be /etc/apache2/apache2.conf):

ScriptAlias "/status/" "/var/www/html/status/status.py"

Finally, restart apache:

sudo service apache2 restart

Publishing Readme to Git Host

If your Git hosting provider (e.g. GitHub) supports symbolic links and reStructuredText rendering, you can just commit your changes and push them to the provider’s remote repository. The Readme will immediately be rendered and updated.

Publishing PDFs

From the LLAMA installation folder, navigate into the documentation folder and generate the PDF:

cd docs
make latexpdf

You can check the quality of your handiwork immediately using xpdf to view the output file:

xpdf build/latex/\*pdf

Publishing HTML Web Pages

From the LLAMA installation folder, navigate into the documentation folder and generate the HTML files:

cd docs
make html

You can run a python server from within docs to view the output:

cd build/html
python -m SimpleHTTPServer

While the python server is running, you can view the documentation webpage by opening any web browser and navigating to the URL of your server (you will have to append the port number, which defaults to 8000 for SimpleHTTPServer). If your server has the URL example.com, then you would want to navigate to example.com:8000. You should then be able to see the webpage.

Troubleshooting LLAMA

Problem: llama run (or some other program that updates files) keeps crashing when trying to generate a skymap or other data generated with MATLAB. The MATLAB log located at logs/llama.matlab.log mentions something about loadjson or savejson (or something else with json in it), like in the below error message:

Undefined function 'loadjson' for input arguments of type 'char'.

Solution: Make sure the JSONLab toolbox is installed in MATLAB. For some reason installation does not persist after the first installation, and a second installation is usually required. You can confirm that JSONLab is installed by opening a MATLAB session and seeing if loadjson is a valid command; if it is, then JSONLab has been successfully installed.

Problem: llama run (or some other program that updates files) keeps crashing when trying to download lvc_initial.fits.gz or some other file located on GraceDB.

Solution: Refresh your LIGO authentication. See the section above on LIGO authentication.

Problem: llama run (or some other program that updates files) keeps crashing when trying to generate a skymap or other data generated with MATLAB; MATLAB claims that files are missing.

Solution: Download the missing ~/multimessenger-pipeline/Neutrinos/Data folder.

Problem: I am running this on a small throwaway server and am out of space, but I don’t want to spend more money on disk space.

Solution: Delete the contents of /var/cache/apt and /usr/share/doc to clear up (probably) around 3GB of data that are unlikely to be of further benefit.

Setting Up the Review Server

These instructions explain how to set up a server for reviewers to test the pipeline on.

Provisioning the review server

Run the following command and enter your admin password (since we’re creating a new user named “reviewer”) as well as info for the new user:

~/multimessenger-pipeline/review-site/provision

This will get everything set up for a new reviewer. In particular, if the reviewer user account has not been made, a command will be run to create the account for you; in this case, you will have to specify a password, accept all other defaults, and then relaunch the .provision script. Now, assuming you have password-based authentication turned on, your reviewers can log in to the server using the username reviewer and the password you specified.

Running the Review

Run:

~/llamatest

This script will run through a few injected test cases (as described on gwhen.com). It will describe what it has done and whether it succeeded; you can confirm the results for yourself by looking in the output directory, ~/.local/share/llama/current_run. The python library used by llamatest is located in ~/multimessenger-pipeline/llama.

Ideas for the Future

This section contains exploratory notes and links on ideas, methods, and features that might be useful for the pipeline in the future.

CVMFS

It’s worth investigating using the CernVM-File System (CernVM-FS, or CVMFS) as a software environment provisioning method. Software and data are represented in a virtual file-system mounted to /cvmfs, and file therefrom are downloaded and cached locally on an as-needed basis, promising fast provisioning times.

LIGO seems to just be starting to support CVMFS at time of writing, while IceCube seems to have mature CVMFS support based on a cursory viewing of their documentation. If software matures in both collaborations to the point where the distributions on CVMFS can dependibly provide all of the needed features, it might be possible to switch over to CVMFS for IceCube and LIGO software.

Advantages

• If other projects support CVMFS, it will be faster to incorporate their tooling into the pipeline using CVMFS.

• We can create very lightweight virtual machine images and docker containers with empty CVMFS caches for archiving, development, and deployment purposes; they will be faster to provision, archive, download, and restart thanks to this reduces size (though in a cluster environment, it would of course cause multiple parallel downloads to CVMFS, which might be a bottleneck, or even worse, an unintentional DDoS attack on CVMFS).

Disadvantages

• Less flexible than having a user-maintained Conda installation.

• root privileges would be necessary for sysadmin, and CVMFS idiosyncracies will likely make it much harder to support heterogeneous platforms in a fault-tolerant way (this is pretty important).