Common-EGSE : Interface Control Document

Each HDF5 file generated by the storage or the data dumper has a group /versions/format_version which contains two attributes, major_version and minor_version. In the following example, we see that the current version number for an N-FEE HDF5 file is 2.6 and for an F-FEE HDF5 file is 3.0. The major version will not change anymore, we fixed it to major_version=2 for N-FEE and major_version=3 for F-FEE. Only the minor version will increase when format changes are introduced in the HDF5 file.

The format_version for the N-FEE HDF5 files is set in the function save_format_version() in the egse.dpu.init.py. For the F-FEE HDF5 files, the format_version is set in the method dump_format_version() of the Data Dumper. As of this writing, the history is as follows:

Since the format and content of the N-FEE and F-FEE HDF5 files are quite different, we have split the description in two sections below. We will go through the different groups, datasets, and attributes of each of these files, explain how to extract the information from the HDF5 file and what the content is. Since some of you are more used to the h5py package we will try to extract and visualize the data using both the egse.h5 as the h5py package. A third visualisation is with the h5tui app, which is a generic HDF5 viewer (a terminal app) that was developed for this project. We will use that with the F-FEE HDF5 files.

This section describes the format and content of an N-FEE HDF5 file. For information about the HDF5 file for the F-FEE, checkout Section 4.2.3.

Let’s take an example file from IAS taken on 7th February 2023. The file is loaded with the h5.get_file() function and we can visualize the top level structure as follows:

We can see that there are five top-level groups and four datasets. This is data taken in external sync mode, so we have four readouts per cycle. The data from each readout is in the groups 0, 1, 2, and 3. These numbers correspond to the frame number. Each of these groups has the following structure:

The data group contains all the SpaceWire packets that have image data, i.e. normal data packets and overscan packets. The data group also has the following attributes that are used to decode the SpaceWire packets into image data arrays. We will describe the data groups into more detail later in this section.

The two datasets in group '/0' contain the timecode and the housekeeping information that is sent on every sync pulse. The timecode dataset contains the timecode itself and the timestamp when this timecode was received by the DPU Processor. Remember the timecode is an integer from 0 to 63. The timecode dataset is an array with one integer element, the timestamp is an attribute of the timecode dataset. The timecode dataset and the timestamp can be visualised as follows.

The raw content of the hk dataset can be shown as follows. The hk dataset has no attributes currently.

There is also a hk_data dataset which contains the housekeeping data that was requested from the FEE after all image data has been transmitted. The difference between the hk and the hk_data is that the former is a HousekeepingPacket object, while the latter is a HousekeepingData object. The reason for this difference is that the hk is a SpW packet that is always sent by the FEE after the timecode, the hk_data is the memory requested that contains the housekeeping information, no SpW packet was every constructed for it (as it was sent in an RMAP read request reply).

The CGSE provides a module to inspect and work with PLATO SpaceWire packets. The above housekeeping packet can be inspected using the HousekeepingPacket class from the egse.spw package:

Inspecting the housekeeping data, can be done as follows, please note that the HousekeepingData class is dependent on the camera type, for the N-FEE, the class is loaded from egse.dpu.npdu.dpu.

Thus far we have explored the following format of the HDF5 file:

We haven’t inspected the versions group yet, it currently contains only one dataset, format_version. This version describes the changes in the HDF5 file with respect to available groups, datasets and attributes. The format version can be accessed as follows.

Up to now, the format versions have changed from 2.0 to 2.6^[1] as follows:

XXXXX: We have decided that the format version of the N-FEE will always be 2 and the format version of the F-FEE will always be 3. This needs to be explained!

Before we dive into the data groups, let’s first inspect the four remaining datasets dpu, fee, obsid and register. The obsid dataset contains the full observation identifier where this HDF5 file belongs to as a bytes object. If the obsid is empty, no observation was running.

The dpu dataset contains DPU Processor specific parameters that are needed to properly process the data. These parameters are available as attributes to this dataset and are mainly used by the FITS generation process.

The fee dataset contains FEE specific parameters that are needed to properly process the data. These parameters are available as attributes to this dataset and are mainly used by the FITS generation process. Currently, the only attribute if the fee/type which can be either 'N-FEE' or 'F-FEE'.

Finally, the register dataset is a Numpy array that is a mirror of the register memory map in the N-FEE at the time of the sync pulse.

The content of the register dataset can be inspected using the RegisterMap class from the CGSE. If you are using a slightly older version of the CGSE, your output might looks different, i.e. not in a nicely formatted table. The content is however the same.

The last group to inspect is the data group which is part of each of the readout groups. The data group contains all the SpaceWire packets that contain the CCD image data. The packets contain the serial prescan, serial overscan, the actual image data and the parallel overscan (if present). From the attributes of the dpu dataset we learned that h_end=2294, v_start=0 and v_end=4539. The h_end attribute defines what is in the row data. The value h_end=2294 means 25 pixels of serial prescan, 2255 pixels of image data, and 15 pixels of serial overscan data. Each packet in the data group is a Numpy array of type uint8, but the actual pixel data is a 16bit integer. The header of a SpW data packet is 10 bytes, so from this information we can calculate that there are 7 lines contained in each packet of length 32140. We also have data packets of 9190 bytes which contain only two rows of data.

We requested 4540 rows (v_end - v_start + 1) which is a full CCD of 4510 rows + 30 rows parallel overscan data. Image data and overscan data are sent in separate packets, so we have 644 + 1 packets of image data and 4 + 1 packets of parallel overscan data.

This gives us a total of 650 packets for one side of the CCD, but since we requested both sides of the CCD (see sensor_sel=3 in the dpu attributes or the register dataset above), we end up with a total of 1300 packets (datasets) in each of the data groups in /0, /1, /2, and /3.

For this section we are going to inspect a few HDF5 files that were created during testing of the F-FEE EM at KU Leuven back in 2024. The tests retrieved CCD image data from each side of the CCDs in a consecutive manner, i.e. no data was retrieved over multiple SpaceWire links simultaneously. The EM didn’t have an AEB4 unit.

The top-level groups of an F-FEE HDF5 file are listed below:

You can see immediately that the top-level groups are quite different from the N-FEE HDF5 file. That is because the concept of cycles is different. Where the N-FEE has a cycle of 25s where the 4 CCDs are read out in 6.25s readout periods, the F-FEE has a cycle time of 2.5s where we read out one half of one CCD over a dedicated SpaceWire link. So, the first group that we see is the AEB1-F which contains the image data for the F-side of the CCD connected to the AEB1 unit. That is currently the bes we can do. We will usually read out 5 frames in 5 different HDF5 files, then switch to another CCD half.

Let’s look into this group into more detail. There is one sub-group data and two datasets hk_deb and hk_aeb. These two datasets are the housekeeping packets that are sent by the F-FEE together with the image data, i.e. when the DEB is in FULL_IMAGE mode and the AEB that was selected is in IMAGE mode.

The data group contains all the SpaceWire packets that have image data, i.e. normal data packets and overscan packets. The data group also has the attributes ccd_side and overscan_lines that are used to decode the SpaceWire packets into image data arrays. We will describe the data groups into more detail later in this section.

The second top-level group is the commands group which contains a description of the commands that were sent right after the sync pulse for this cycle. The commands are numbered starting from zero and have an attribute duration which gives the total time that this command needed to execute (in seconds). The h5tui shows the commands with syntax highlighting. We see that six commands were sent during this cycle, sync_register_map, deb_read_hk and four aeb_read_hk. We find back the result of these commands in the other groups.

The Register Map that was synchronised is stored in the top-level register group as numpy array. You will have to convert this array into a RegisterMap object to inspect it. If you use the rich package to print this, you will get a nice table with all register parameters and their attributes and values. The RegisterMap is synchronised on every cycle right after the reception of the timecode.

The other five commands have read the housekeeping data of the DEB and the four AEBs. The result of these commands is stored in the hk-data group. As with the register map, you will find these HK data are stored as a numpy array and need to be converted into a HousekeepingData object. If you use the h5tui app, the HK data is nicely presented in a table.

There are a few other top-level groups and datasets that we will discuss briefly here.

This concludes our trip trough the F-FEE HDF5 file. If you are running the h5tui tool in a terminal that can handle enough colors, you can inspect the image data also from within this tool by selecting the data group. In the footer you can see which keys to use for zooming and panning. Pressing 'b' will zoom into the area with the brightest pixel, pressing '0' will bring you back to a properly zoomed figure.

Sometimes, we get a problem in receiving image data packets and the packets we get are corrupt. Those packets are not saved as part of the data group in the AEB group, but are saved as plain vanilla SpaceWirePacket objects. You can inspect these packets with the h5tui tool. It might be useful to determine their sequence counter, frame counter, etc.

For each device we need to add all HK parameters to the TM dictionary. For each of these parameters you need to add one line with the following information (in the designated columns):

Since the TM dictionary grows longer and longer, the included devices/processes are ordered as follows (so it is easier to find back the telemetry parameters that apply to your TH):

The correct (i.e. EGSE-conform) naming of the telemetry should be taken care of in the get_housekeeping method of the device protocols.

The Synoptics Manager groups a pre-defined set of HK values in a single file. It’s not the original EGSE-conform names that are use in the synoptics, but names with the prefix GSYN_. The following information is comprised in the synoptics:

For the first type of telemetry parameters, their original EGSE-conform name should be put into the column CAM EGSE mnemonic, as they are not TH-specific.

The second type of telemetry parameters is measured with TH-specific devices. The original TH-specific EGSE-conform name should go in the column Origin of synoptics at ....

The translation tables that were mentioned in the introduction, can be created by the read_conversion_dict function in egse.hk. It takes the following input parameters:

To apply the actual translation, you can use the convert_hk_names function from egse.hk, which takes the following input parameters:

When you want to include HK of your devices, you need to take the following actions:

Please, do not introduce new synoptics without further discussion!

TBW

The N-FEE sends out a housekeeping packet on every sync pulse, right after it sends the timecode. This housekeeping packet contains temperature sensor values, voltages, currents, some status parameters and error codes. Sensor values, voltages and currents are raw measures and are converted into calibrated values by the n_fee_hk process. The conversion is defined in the Setup for that camera (in the setup.camera.fee.calibration entry).

All raw and calibrated housekeeping values are stored in the N-FEE-HK CSV file. There are currently about 154 columns in that file. The timestamp of each row is the reception time of the HK packet by the DPU Processor.

Actually, since release 2023.20.0+CGSE of the Common-EGSE, there are two rows of HK values per timecode. The DPU Processor requests all HK data from the N-FEE immediately after receiving the image data from the N-FEE and before sending any RMAP requests. Even if no image data is generated, e.g. in STANDBY or DUMP mode, the additional updated housekeeping is requested. This additional information is needed because the housekeeping contains information, especially error codes, on the readout and data transfer. Also interesting to know is that the first housekeeping packet sent by the N-FEE contains information about the previous readout, while the updated HK data that was requested from the N-FEE contains information about the current readout.

TBW