Pipeline
========

.. warning:: 

   The following guide explains how to install, setup, and run the LESIA pipeline as an integrated part of PLATOnium. **A (Linux) bash shell is required to install this setup** since this tool should be used on a computing cluster where Linux is the standard. Mac users may have to adapt especially the ``setup.py`` script.

   In order to have a frozen functional setup between PlatoSim/PLATOnium and the LESIA pipeline, the lastest tested versions are:

   **PlatoSim develop branch:**

   .. code-block:: shell

      git checkout 3.6.0-225-g552813f9
   
   **LESIA SVN pipeline**

   .. code-block:: shell

      svn checkout 5420
         
.. raw:: html

   <hr>





   
.. _pipeline_run:

   
*Installation*
-------------- 
      
The LESIA pipeline is a SVN (subversion) repository, hence, if not installed, first install SVN

.. code-block:: shell

   sudo apt update		   
   sudo apt install subversion -y

Confirm the installation by viewing the version: ``svn --version``.

Next create a directory called ``plato``. Users are recommended to create this folder in the ``$PLATO_PROJECT_HOME`` directory. Developers are on the other hand encouraged to place this folder in the same (software) directory to where ``PlatoSim3`` is located. Hence, type:

.. code-block:: shell

   mkdir </path/to/plato>
   cd </path/to/plato>

Now download the SVN repository by providing your ``username`` and ``password`` (contact the LESIA team for the credentials): 

.. code-block:: shell

   svn checkout https://version-lesia.obspm.fr/repos/PLATO/algorithms/ --username <username> --password <password>
                
If you are a user, first download the script (`setup.sh script from Github <https://github.com/IvS-KULeuven/PlatoSim3/blob/develop/setup.sh>`_) and place it in your ``$PLATO_PROJECT_HOME`` directory. Being a developer, this script already exists.

To setup and install the pipeline, two arguments need to be parsed; the first being the path to your PlatoSim working directory (``</path/to/plato_workdir>``) and the second being the path to your newly created **plato** folder (``</path/to/plato>``). Thus, simply run:
   
.. code-block:: shell

   ./setup </path/to/plato_workdir> </path/to/plato>

This script will create a ``.bash_profile`` file in the ``$PLATO_PROJECT_HOME`` directory and:

- Export the environment variable ``$PLATO_PROJECT_HOME``
- Export the environment variable ``$PLATO_WORKDIR``
- Export the environment variable ``$PLATO``
- Export the environment variable ``$PYTHONPATH``
- Create the directory tree needed to install the LESIA pipeline
- Installs the LESIA pipeline
- Make the LESIA pipeline scripts globally executable
- Make the PLATOnium scripts globally executable

If you want to change your working directory at a later stage, simply run ``setup.sh`` again. The script will automatically detect that the LESIA pipeline is already installed. You can force a new install of the LESIA pipeline by parsing the argument ``--reinstall``.

.. warning::
   
   In case of installation errors, we recomment to pin down the issue by installing each dependency one-by-one. Have a look inside the file ``$PLATO/algorithms/Makefile``. This file contains the instructions for ``make install`` and ``make clean``.

   Moreover, well known problems with the installation can be found in the :ref:`Troubleshooting <platonium_pipeline_troubleshooting>` section below.

   
.. admonition:: Update the LESIA pipeline:

   To update the SVN repository, simply go to ``cd $PLATO/algortihms`` and do:

   .. code-block:: shell

      svn update
      sudo make

   To checkout a specific revision use:

   .. code-block:: shell

      svn checkout <revision>
      
.. raw:: html

   <hr>





   
.. _pipeline_run:

*Run examples*
--------------

Compared to the previous tutorial on how to run ``platonium``, we now only have to parse the ``--pipeline --sample <plato_sample>`` to activate the LESIA pipeline. If ``P1`` is parsed, the on-ground pipeline is activated, and if ``P5`` is parsed, the on-board pipeline is activated. E.g. say that we already made two seperate stellar catalogues of respectively P1 and P5 sample stars using ``picsim``. A simple test of 30 exposures for each chain of the pipeline looks like the following:

.. code-block::

   platonium 1 1 1 1 --project <project_name> --pipeline --sample P1 --nexp 30
   platonium 1 1 1 1 --project <project_name> --pipeline --sample P5 --nexp 30

Note that a lot of information is printed to bash. When running on a computing cluster, this behavior is typically not desired and hence the flag ``-v 0`` avoid printing to bash.

.. raw:: html

   <hr>





   
.. _pipeline_output:

*Output files*
--------------

During a simulation, PLATOnium creates three folders called ``reduced``, ``microscan``, ``P1`` (or ``P5`` depending on sample parsed). By default during the run a lot of files are created and stored in ``microscan`` and ``P1``, but at end simulation, the final files are stored into ``reduced``. Note that these foder contains a tree of subfolders to keep each simulation isolated when running in parallel. A standard filename is used for each data product, e.g. the light curve file ``000000001_Ncam1.1_Q1.ftr`` refers to the star ID (``000000001``), the N-CAM camera-group no. and camera no. (``Ncam1.1``), and the quarter (``Q1``). 

**Unique files returned after a P1 sample run:**

* Final extracted P1 sample data with the following columns (``*.ftr``):
  
  - ``time``     : [second] Time points with zero-point at mission BOL
  - ``flux``     : [electron] Extracted flux
  - ``cx``       : [pixel] X centroid pixel within 6x6 pixel subfield 
  - ``cy``       : [pixel] Y centroid pixel within 6x6 pixel subfield
  - ``bg``	 : [electron] Background flux
  - ``flux_err`` : [electron] Flux error    
  - ``cx_err``   : [pixel] X centroid pixel error
  - ``cy_err``   : [pixel] Y centroid pixel error
  - ``bg_err``   : [electron] Background flux error
  - ``chi2``  	 : Chi-sqaured of PSF fit
  - ``iter``     : Iterations before convergecne for PSF fitting 


**Unique files returned after a P5 sample run:**

* Final extracted P5 sample data with the following columns (``*.ftr``): 

  - ``time``  		: [second] Time points with zero-ponint at mission BOL
  - ``flux``        	: [electron] Extracted and corrected flux
  - ``xc``        	: [pixel] X centroid pixel within 6x6 pixel subfield
  - ``yc``  		: [pixel] Y centroid pixel within 6x6 pixel subfield
  - ``flux_cor`` 	: [norm.] Flux model for jitter/drift correction

* Mask-update file for each target star (Marchiori+2019; ``*.fits``).

* Stellar Polution Ratio (SPR) information of target star as per Marchiori+2019 (``*.spr``).

  
**Files indentical for P1 and P5 sample:**

* An overview table (in feather format) of the simulation (``*.table``):

  - ``ID``        : Star ID of nine digits
  - ``PIC``       : PIC name of target
  - ``ra``        : [degree] ICRS Right Ascension 
  - ``dec``       : [degree] ICRS Declination
  - ``mag``       : PALTO passband magnitude P
  - ``group``     : Camera-group ID
  - ``camera``    : Camera ID
  - ``quarter``   : Mission quarter number
  - ``ccd``       : CCD ID for star (subfield) location
  - ``xCCD``      : [pixel] X centroid pixel coordinate of full frame CCD of first image
  - ``yCCD``      : [pixel] Y centroid pixel coordinate of full frame CCD of first image
  - ``rOA``       : [degree] Radial distance to optica axis
  - ``xFP``       : [millimeter] X focal plane coordinate of first image
  - ``yFP``       : [millimeter] Y focal plane coordinate of first image
  - ``ncon``      : Number of contaminants in subfield

* Information about the PSF inversion (``*.invert``). For more information see: PLATO-PL-LESIA-TN-0069 and PLATO-PL-LESIA-TN-0070.

.. raw:: html

   <hr>






   
.. _platonium_pipeline_troubleshooting:

*Troubleshooting*
-----------------



.. _platonium_pipeline_troubleshooting_python:

Installation errors: Python
...........................

**pymt64:** The package ``pymt64`` can sometimes cause porblems. A well known issue for this pacakage is the following error:

.. code-block:: shell

   import pymt64
   File "pymt64.pyx", line 1, in init pymt64
   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

This can be solved by re-installing the library:

.. code-block:: shell

   pip uninstall pymt64
   pip install --no-cache-dir pymt64




.. _platonium_pipeline_troubleshooting_makefile:
   
Installation errors: Makefile
.............................

A few obstacles can happen writing a code assuming full sudo rights (and assuming that apt get is available). In order to successfully install the LESIA pipeline on a cluster, the following ajustments may need to be inforced:

**Disabled environment:** Make sure that the ``$PLATO`` path is defined. If added to your ``.bash_profile`` file this path should automatically be loaded every time you enter a compute node on any machine/cluster. However, it occured that it was not and the installation will link to a wrong location and fail.

**aswsim3:** This library ``aswsim3`` sometimes cause problems and since we do not need to PLATOnium we can simply skip this library. However, ``aswsim3`` depends on the library ``platosimlib``, hence, simply replace ``aswsim3`` with ``platosimlib`` in the section ``install: all`` listed within in ``Makefile``. Note the problem is typically that GLS packages cannot be found here, like:

.. code-block:: shell
		
   ../src/test_gsl.c:2:24: fatal error: gsl_matrix.h: No such file or directory
   #include <gsl_matrix.h>


**OpenBLAS vs. BLAS/LAPACK:** If using OpenBLAS as a BLAS (and LAPACK) implementation the ``-lblas`` and ``-llapack`` link arguments would be appropriate when linking to the reference BLAS/LAPACK libraries from NetLib, which are typically called ``libblas.so`` and ``liblapack.so``. However, with OpenBLAS both are in the same shared library called ``libopenblas.so``.

1. The first installation error you may encounter is thus from the library ``WP/326100/spline2dbase``. This library uses the ``setup.py`` script within for the installation which link to the ``-lblas`` and ``-llapack``. Thus, if using OpenBLAS you will need to replace ``-lblas`` and ``-llapack`` with ``openblas`` in the following line:

   .. code-block:: shell

      libraries=["m","blas","lapack"],

   to

   .. code-block:: shell
		
      libraries=["m","openblas"],
   
   For the rest of the libraries you need to look into the ``Makefile`` of each package.

2. Within the main installation Makefile (``$PLATO/algorithms/Makefile``) the inversion modules named ``WP/32100/voxel/..`` link to the openblas libray. If you get a message that ``-llapack not found`` then a potential solution is to remove this flag completely since it is already included within openblas. Hence remove ``-llapack`` from the following line in each file you encounter this error:

   .. code-block:: shell

      LDLIBS = -lcvxs -lm -llapack -lopenblas -lpthread

   
**Man pages:** Since the user do not generally have sudo right on a computing cluster the inversion module tries to install the man pages for the packges ``WP/32100/voxel/..``. This can result in the following error:

.. code-block:: shell
  
   warning: failed to load external entity "/usr/share/xml/docbook/stylesheet/nwalsh/html/docbook.xsl"
   cannot parse /usr/share/xml/docbook/stylesheet/nwalsh/html/docbook.xsl

These are really not needed for our computations and hence

1. Remove the ``.1`` and ``.txt`` files from ``all :`` that will be installed
2. Comment out the line ``install -m 644 combine-mat.1 $(PLATOMAN1)/..``
3. Run ``make install`` again

   
.. _platonium_pipeline_troubleshooting_executables:
  
Run error: Executables missing
..............................

Sometimes the following scripts: ``combine-mat``, ``invert-rls``, ``spline2real``, and ``find-amp`` cannot be found globally (typically true on a computing cluster by the local node you are running simulations on). By default these scripts are installed into your ``$PLATO/bin`` folder, however, the script ``setup.sh`` should take care of copying these files to your Conda environment **bin** folder.

If the above script are not directly executeable from bash (e.g. ``combine-mat -h``) then the PLATOnium setup script most likely failed. Thus, simply copy all of these files to your Conda environment **bin** folder. You can find this location by first activating your Conda environment and thereafter type ``which python``. Copy the files into the **bin** folder and not the **bin/python** folder!