Pipeline

Warning

The following guide explains how to install, setup, and run the LESIA pipeline as an integrated part of PLATOnium. A (Linux) bash shell is required to install this setup since this tool should be used on a computing cluster where Linux is the standard. Mac users may have to adapt especially the setup.py script.

In order to have a frozen functional setup between PlatoSim/PLATOnium and the LESIA pipeline, the lastest tested versions are:

PlatoSim develop branch:

git checkout 3.6.0-225-g552813f9

LESIA SVN pipeline

svn checkout 5420

Installation

The LESIA pipeline is a SVN (subversion) repository, hence, if not installed, first install SVN

sudo apt update
sudo apt install subversion -y

Confirm the installation by viewing the version: svn --version.

Next create a directory called plato. Users are recommended to create this folder in the $PLATO_PROJECT_HOME directory. Developers are on the other hand encouraged to place this folder in the same (software) directory to where PlatoSim3 is located. Hence, type:

mkdir </path/to/plato>
cd </path/to/plato>

Now download the SVN repository by providing your username and password (contact the LESIA team for the credentials):

svn checkout https://version-lesia.obspm.fr/repos/PLATO/algorithms/ --username <username> --password <password>

If you are a user, first download the script (setup.sh script from Github) and place it in your $PLATO_PROJECT_HOME directory. Being a developer, this script already exists.

To setup and install the pipeline, two arguments need to be parsed; the first being the path to your PlatoSim working directory (</path/to/plato_workdir>) and the second being the path to your newly created plato folder (</path/to/plato>). Thus, simply run:

./setup </path/to/plato_workdir> </path/to/plato>

This script will create a .bash_profile file in the $PLATO_PROJECT_HOME directory and:

  • Export the environment variable $PLATO_PROJECT_HOME

  • Export the environment variable $PLATO_WORKDIR

  • Export the environment variable $PLATO

  • Export the environment variable $PYTHONPATH

  • Create the directory tree needed to install the LESIA pipeline

  • Installs the LESIA pipeline

  • Make the LESIA pipeline scripts globally executable

  • Make the PLATOnium scripts globally executable

If you want to change your working directory at a later stage, simply run setup.sh again. The script will automatically detect that the LESIA pipeline is already installed. You can force a new install of the LESIA pipeline by parsing the argument --reinstall.

Warning

In case of installation errors, we recomment to pin down the issue by installing each dependency one-by-one. Have a look inside the file $PLATO/algorithms/Makefile. This file contains the instructions for make install and make clean.

Moreover, well known problems with the installation can be found in the Troubleshooting section below.

Update the LESIA pipeline:

To update the SVN repository, simply go to cd $PLATO/algortihms and do:

svn update
sudo make

To checkout a specific revision use:

svn checkout <revision>

Run examples

Compared to the previous tutorial on how to run platonium, we now only have to parse the --pipeline --sample <plato_sample> to activate the LESIA pipeline. If P1 is parsed, the on-ground pipeline is activated, and if P5 is parsed, the on-board pipeline is activated. E.g. say that we already made two seperate stellar catalogues of respectively P1 and P5 sample stars using picsim. A simple test of 30 exposures for each chain of the pipeline looks like the following:

platonium 1 1 1 1 --project <project_name> --pipeline --sample P1 --nexp 30
platonium 1 1 1 1 --project <project_name> --pipeline --sample P5 --nexp 30

Note that a lot of information is printed to bash. When running on a computing cluster, this behavior is typically not desired and hence the flag -v 0 avoid printing to bash.

Output files

During a simulation, PLATOnium creates three folders called reduced, microscan, P1 (or P5 depending on sample parsed). By default during the run a lot of files are created and stored in microscan and P1, but at end simulation, the final files are stored into reduced. Note that these foder contains a tree of subfolders to keep each simulation isolated when running in parallel. A standard filename is used for each data product, e.g. the light curve file 000000001_Ncam1.1_Q1.ftr refers to the star ID (000000001), the N-CAM camera-group no. and camera no. (Ncam1.1), and the quarter (Q1).

Unique files returned after a P1 sample run:

  • Final extracted P1 sample data with the following columns (*.ftr):

    • time : [second] Time points with zero-point at mission BOL

    • flux : [electron] Extracted flux

    • cx : [pixel] X centroid pixel within 6x6 pixel subfield

    • cy : [pixel] Y centroid pixel within 6x6 pixel subfield

    • bg : [electron] Background flux

    • flux_err : [electron] Flux error

    • cx_err : [pixel] X centroid pixel error

    • cy_err : [pixel] Y centroid pixel error

    • bg_err : [electron] Background flux error

    • chi2 : Chi-sqaured of PSF fit

    • iter : Iterations before convergecne for PSF fitting

Unique files returned after a P5 sample run:

  • Final extracted P5 sample data with the following columns (*.ftr):

    • time : [second] Time points with zero-ponint at mission BOL

    • flux : [electron] Extracted and corrected flux

    • xc : [pixel] X centroid pixel within 6x6 pixel subfield

    • yc : [pixel] Y centroid pixel within 6x6 pixel subfield

    • flux_cor : [norm.] Flux model for jitter/drift correction

  • Mask-update file for each target star (Marchiori+2019; *.fits).

  • Stellar Polution Ratio (SPR) information of target star as per Marchiori+2019 (*.spr).

Files indentical for P1 and P5 sample:

  • An overview table (in feather format) of the simulation (*.table):

    • ID : Star ID of nine digits

    • PIC : PIC name of target

    • ra : [degree] ICRS Right Ascension

    • dec : [degree] ICRS Declination

    • mag : PALTO passband magnitude P

    • group : Camera-group ID

    • camera : Camera ID

    • quarter : Mission quarter number

    • ccd : CCD ID for star (subfield) location

    • xCCD : [pixel] X centroid pixel coordinate of full frame CCD of first image

    • yCCD : [pixel] Y centroid pixel coordinate of full frame CCD of first image

    • rOA : [degree] Radial distance to optica axis

    • xFP : [millimeter] X focal plane coordinate of first image

    • yFP : [millimeter] Y focal plane coordinate of first image

    • ncon : Number of contaminants in subfield

  • Information about the PSF inversion (*.invert). For more information see: PLATO-PL-LESIA-TN-0069 and PLATO-PL-LESIA-TN-0070.

Troubleshooting

Installation errors: Python

pymt64: The package pymt64 can sometimes cause porblems. A well known issue for this pacakage is the following error:

import pymt64
File "pymt64.pyx", line 1, in init pymt64
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

This can be solved by re-installing the library:

pip uninstall pymt64
pip install --no-cache-dir pymt64

Installation errors: Makefile

A few obstacles can happen writing a code assuming full sudo rights (and assuming that apt get is available). In order to successfully install the LESIA pipeline on a cluster, the following ajustments may need to be inforced:

Disabled environment: Make sure that the $PLATO path is defined. If added to your .bash_profile file this path should automatically be loaded every time you enter a compute node on any machine/cluster. However, it occured that it was not and the installation will link to a wrong location and fail.

aswsim3: This library aswsim3 sometimes cause problems and since we do not need to PLATOnium we can simply skip this library. However, aswsim3 depends on the library platosimlib, hence, simply replace aswsim3 with platosimlib in the section install: all listed within in Makefile. Note the problem is typically that GLS packages cannot be found here, like:

../src/test_gsl.c:2:24: fatal error: gsl_matrix.h: No such file or directory
#include <gsl_matrix.h>

OpenBLAS vs. BLAS/LAPACK: If using OpenBLAS as a BLAS (and LAPACK) implementation the -lblas and -llapack link arguments would be appropriate when linking to the reference BLAS/LAPACK libraries from NetLib, which are typically called libblas.so and liblapack.so. However, with OpenBLAS both are in the same shared library called libopenblas.so.

  1. The first installation error you may encounter is thus from the library WP/326100/spline2dbase. This library uses the setup.py script within for the installation which link to the -lblas and -llapack. Thus, if using OpenBLAS you will need to replace -lblas and -llapack with openblas in the following line:

    libraries=["m","blas","lapack"],

    to

    libraries=["m","openblas"],

    For the rest of the libraries you need to look into the Makefile of each package.

  2. Within the main installation Makefile ($PLATO/algorithms/Makefile) the inversion modules named WP/32100/voxel/.. link to the openblas libray. If you get a message that -llapack not found then a potential solution is to remove this flag completely since it is already included within openblas. Hence remove -llapack from the following line in each file you encounter this error:

    LDLIBS = -lcvxs -lm -llapack -lopenblas -lpthread

Man pages: Since the user do not generally have sudo right on a computing cluster the inversion module tries to install the man pages for the packges WP/32100/voxel/... This can result in the following error:

warning: failed to load external entity "/usr/share/xml/docbook/stylesheet/nwalsh/html/docbook.xsl"
cannot parse /usr/share/xml/docbook/stylesheet/nwalsh/html/docbook.xsl

These are really not needed for our computations and hence

  1. Remove the .1 and .txt files from all : that will be installed

  2. Comment out the line install -m 644 combine-mat.1 $(PLATOMAN1)/..

  3. Run make install again

Run error: Executables missing

Sometimes the following scripts: combine-mat, invert-rls, spline2real, and find-amp cannot be found globally (typically true on a computing cluster by the local node you are running simulations on). By default these scripts are installed into your $PLATO/bin folder, however, the script setup.sh should take care of copying these files to your Conda environment bin folder.

If the above script are not directly executeable from bash (e.g. combine-mat -h) then the PLATOnium setup script most likely failed. Thus, simply copy all of these files to your Conda environment bin folder. You can find this location by first activating your Conda environment and thereafter type which python. Copy the files into the bin folder and not the bin/python folder!