Pipeline ======== Welcome to the PLATO Ecosystem: A Docker wrapper for PlatoSim3 and the LESIA L1 pipeline. The following guide explains how to install, setup, and run the LESIA L1 pipeline as an integrated part of PLATOnium. Courtesy goes to `Réza Samadi (LESIA) `_, who led the code development for the pipeline, and to `James McCormac (UW) `_, who developed the Docker Ecosystem. .. _pipeline_setup: *Initial setup* --------------- * Install docker and docker-compose following the instructions for your host OS. * Clone the `GitHub docker ecosystem `_ repository to your server * Inside the ``docker_ecosystem`` folder clone the ``PlatoSim3`` repo (checkout whatever branch you currently want to use, e.g. develop) * Inside the ``docker_ecosystem`` folder create a folder called ``algos`` * Inside the ``algos`` folder clone the LESIA L1 pipeline ``common`` repository * Speak to KUL developers, LESIA developers, and James McCormac for access if required to PlatoSim3, L1 pipeline, and Ecosystem, respectively. The software packages are cloned into this parent level directory to avoid passing git credentials into the Docker images. Once cloned the codes are copied into the Docker image as part of the setup. See image below for a schematic of the system. .. image:: ../figures/Docker.png :align: center :width: 750 .. _pipeline_docker: *Installation* -------------- .. _pipeline_docker_build: Building the Docker image ......................... * From the ``docker_ecosystem`` parent level folder do the following * Run ``./install.sh`` * An Ubuntu 20.04 image will be created with the following actions: - Python 3.9 - Sets up a non-root user etc - Linux libraries for PlatoSim3 and L1 the pipeline - Python modules for PlatoSim3 and L1 the pipeline - PlatoSim3 and the L1 pipeline themselves * Building an image from scratch with no caching takes about 30 min. .. _pipeline_docker_storage: Configuring simulation storage area on host ........................................... Edit the ``docker-compose.yml`` file, adding a path on your host machine where you'd like to save the data. Specifically, edit the line in the ``volumes:`` section to map ``/path/on/host/:/host_dir``. Docker will mount this area when the container stars and results will persist when the container is stopped. .. _pipeline_docker_startstop: Starting or stopping a container ................................ * Execute the ``run.sh`` script to spin up a container in interactive mode * This uses ``docker-compose`` to mount the storage area inside a container * Simulations can then be run inside the container as normal (see below) * Type ``exit`` to quit the container (as if it was a normal terminal) .. _pipeline_docker_updating: Updating software in the Docker image ..................................... If one of the three software packages is updated, simply pull the latest code into the ecosystem folder and rerun the ``./install.sh`` command. .. _pipeline_docker_prune: Pruning Docker resources ........................ If you use Docker to build many images it's advisable to run: .. code-block:: shell docker system prune -a occasionally to free up some resources. Docker caches image layers to increase build speed but after a while those layers become stale as images are updated. .. raw:: html
.. _pipeline_example: *Run example* ------------- In this example we show hot to run the PlatoSim + L1 Pipeline in the container. Platonium is a wrapper around both PlatoSim and the L1 pipeline. Below is the current usage for platonium and an example command to simulate 1 quarter of data for a given star on a particular camera including some supplied variability. Platonium requires the normal PlatoSim ``inputfile.yaml`` along with a catalog from ``picsim``, any variability signals (e.g. from ``varsim``) and instrument specific configurations for multiple cameras from ``payload``. Please see the full platonium documentation for a full description of the inputfiles. Note, ``picsim``, ``varsim`` and ``payload`` can be run inside the container environment also. Platonium expects these input files in the following directory: .. code-block:: shell /host_dir//input where ```` is described below and ``/host_dir`` is the folder mounted from the host OS in the ``docker-compose.yml`` file above. The following command can be run inside the container to produce a quarter long simulation of a single star (number 46) from camera group 1 camera 1 from quarter 23 and assuming the P5 sample. We also inject stellar/transit signals using the ``--varfile``. The project ``kul20`` corresponds to the KUL technical note 20 simulation settings, therefore for this run the inputs should be stored in ``/host_dir/kul20/input``. .. code-block:: shell platonium 46 1 1 23 --project kul20 --sample P5 --pipeline --varfile /host_dir/varsource/P5/varsource_000000046.txt -v 3 -w Platonium runs PlatoSim and analyses the imagettes. Outputs (photometry etc) are stored in ``/host_dir/kul20/output/reduced/P5/000000046/``. Platonium also produces a summary of the processing time when complete. .. raw:: html
.. _pipeline_output: *Output files* -------------- .. raw:: html :file: pipeline.html