Always use CamelCase (Python uses CapWords) for class names. When using acronyms, keep them all UPPER case.

Good names are: Observation, CalibrationFile, MetaData, Message, ReferenceFrame, URLParser.

A function or a method does something (and should only do one thing, SRP=Single Responsibility Principle), it is an action, so the name should reflect that action.

Always use lowercase words separated with underscores.

Good names are: get_time_in_ms(), get_commanding_port(), is_connected(), parse_time(), setup_mask().

When working with legacy code or code from another project, names may be in camelCase (with the first letter a lower case letter). So we can in this case use also getCommandPort() or isConnected() as method and function names.

Use the same naming convention as functions and methods, i.e. lowercase with underscores.

Good names are: key, value, user_info, model, last_value

Bad names: NSegments, outNoise

Take care not to use builtins: list, type, filter, lambda, map, dict, …​

Private variables (for classes) start with an underscore: _name or _total_n_args.

In the same spirit as method and function names, the variables can also be in camelCase for specific cases.

Use ALL_UPPER_CASE with underscores for constants. Use constants always within a name space, not globally.

Good names: MAX_LINES, BLACK, YELLOW, ESL_LINK_MODE_DISABLED

Use simple words for modules, preferably just one word like datasets or commanding or storage or extensions. If two words are unavoidable, just concatenate them, like multiprocessing or sampledata or testdata. If needed for readability, use an underscore to separate the words, e.g. image_analysis.

Be careful that you do not name any modules the same as a module in the Python standard library. This can result in strange effects and may result in an AttributeError. Suppose you have named a module math in the egse directory and it is imported and used further in the code as follows:

This will result in the following runtime error:

Of course this is an obvious example, but it might be more obscure like e.g. in this GitHub issue: 'module' object has no attribute 'Cmd'.

Exceptions shouldn’t really be caught unless you have a really good plan for how to recover. If you have no such plan, let the exception propagate to a higher level in your software until you know how to handle it. In your own code, try to avoid raising an exception in the first place, design your code and classes such that you minimise throwing exceptions.

If some code path simply must broadly catch all exceptions (i.e. catch the base Exception class) — for example, the top-level loop for some long-running persistent process — then each such caught exception must write the full stack trace to the log, along with a timestamp. Not just the exception type and message, but the full stack trace. This can easily be done in Python as follows:

This will log the exception and stacktrace with logging level ERROR.

For all other except clauses — which really should be the vast majority — the caught exception type must be as specific as possible. Something like KeyError, ConnectionTimeout, etc. That should be the exception that you have a plan for, that you can handle and recover from at this level in your code.

Use the try/except blocks around code that can potentially generate an exception and your code can recover from that exception.

Any resources such as open files or connections can be closed or cleaned up in the finally clause. Remember that the code within finally will always be executed, regardless of an Exception is thrown or not. In the example below, the function checks if there is internet connection by opening a socket to a host which is expected to be available at all times. The socket is closed in the finally clause even when the exception is raised.

When you use a with statement in your code, resources are automatically closed when an Exception is thrown, but the Exception is still thrown, so you should put the with block inside a try/except block.

As of Python 3 the socket class can be used as a context manager. The example above can thus be rewritten as follows:

Another example from our own code base shows how to handle a Hexapod PUNA Proxy error. Suppose you want to send some commands to the PUNA within a context manager as follows:

When you execute the above code and the PUNA control server is not active, the Proxy will not be able to connect and the context manager will raise a ConnectionError.

So, the with .. as statement should be put into a try .. except clause. Since we probably cannot handle this exception at this level, we only log the exception and re-raise the connection error.

In languages like the C programming language is it custom to return error codes or -1 as a return code from a function to indicate a problem has occurred. The main drawback here is that your code, when calling such a function, must always check it’s return codes, which is often forgotten or ignored.

In Python, throw exceptions instead of returning an error code. Exceptions ensure that failures do not go unnoticed because the calling code didn’t check a return value.

The question basically is if we should check for a common condition without possibly throwing an exception. Python does this different than other languages and prefers the use of exceptions. There are two different opinions about this, EAFP and LBYL. From the Python documentation:

This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.

This coding style explicitly tests for pre-conditions before making calls or lookups. This style contrasts with the EAFP approach and is characterized by the presence of many if statements. In a multi-threaded environment, the LBYL approach can risk introducing a race condition between “the looking” and “the leaping”.

Consider the following two cases:

If you expect that 90% of the time your code will just run as expected, use the try/except approach. It will be faster if exceptions really are exceptional. If you expect an abnormal condition more than 50% of the time, then using if is probably better.

In other words, the method to choose depends on how often you expect the event to occur.

Sometimes you just want to do something and rethrow the same Exception. This is easy in Python as shown in the following example.

In some cases, it is best to have the stacktrace printed out with the logging message. I’ve include the exc_info=True in the example.

It is nearly free to set up a try/except block (an exception manager), while an if statement always costs you.

Bear in mind that Python internally uses exceptions frequently. So, when you use an if statement to check e.g. for the existence of an attribute (the hasattr() method), this builtin function will call getattr(obj, name) and catch AttributeError. So, instead of doing the following:

you can better use the try/except.

As a general rule, try to use builtin exceptions from Python, especially ValueError, IndexError, NameError, and KeyError. Don’t invent your own 'parameter' or 'arguments' exceptions if the cause of the exception is clear from the builtin. The hierarchy of Exceptions can be found in the Python documentation at Builtin-Exceptions > Exception Hierarchy.

When the connection with a builtin exception is not clear however, create your own exception from the Exception class.

In some situations you might want to group many possible sources of internal errors into a single exception with a clear message. For example, you might want to write a library module that throws its own exception to hide the implementation details, i.e. the user of your library shouldn’t have to care which extra libraries you use to get the job done.

Since this will hide the original exception, if you throw your own exception, make sure that it contains every bit of information from the originally caught exception. You’ll be grateful for that when you read the log files that are send to you for debugging.

The example below is taken from the actual source code. This code catches all kinds of exceptions that can be raised when connecting to a hardware device over a TCP socket. The caller is mainly interested of course if the connection could be established or not, but we always include the information from the original exception with the raise..from clause.

Use assertions only to check for invariants. Assertions are meant for development and should not replace checking conditions or catching exceptions which are meant for production. A good guideline to use assert statements is when they are triggering a bug in your code. When your code assumes something and acts upon the assumption, it’s recommended to protect this assumption with an assert. This assert failing means your assumption isn’t correct, which means your code isn’t correct.

Another example is:

This is a naming convention thing…​ names of user defined sub-classes of Exception should end with Error.

TBW

Generally, you should not log exceptions at lower levels, but instead throw exceptions and rely on some top level code to do the logging. Otherwise, you’ll end up with the same exception logged multiple times at different layers in your application.

Some of the explanations were taken shamelessly from the following resources:

The docstring shall contain all information that the user of the function, method, class or module needs to use it. The user in this case can be a developer or a normal user of a Python function. Try to adapt the type and detail of the information in the docstring accordingly.

Do not describe how the functionality is implemented, instead describe what functionality is provided and how it can be used.

We will provide an API documentation that contains all docstrings of all classes, functions, methods and modules that are part of the CGSE. If you are using a Python IDE, use the available functionality to visualize the docstring information of a function. For PyCharm e.g. this is the 'F1' key when the cursor is on the function name.

If you are in the IPython REPL, you can type the function name followed by q question mark:

TBW

The configuration manager is a core service…​

The storage manager is responsible for storing the following data:

We have some problems with performance in reading out the N-FEE when we request both sides of a CCD. The reason for that is probably a series of events, such as waiting for a busy storage manager, garbage collection, data processing, etc. When starting to develop the code for the fast camera, we want to avoid these performance problems as much as possible and we made a number of decisions that are explained below.

So, we started out designing the new F-DPU Processes.

There are two main problems that we identified with the Data Storage. (1) The Storage Manager forms a bottleneck handling all data that is generated by the system, not only writing science data to the HDF5 files, but also handling (and duplicating) all housekeeping and auxiliary data that is generated by all the devices used during the test. (2) Using the request-reply (REQ-REP) protocol, the process that generates the data and sends it to the storage manager has to wait for a reply from the Storage Manager with a confirmation of the data being written properly. This might lead to a delay when the Storage manager is very busy writing information to different files and formats.

We decided to keep the functionality of the Storage Manager as it is, don’t touch it since it is still used for all N-CAM activities. In addition, we will build a new process that will only handle the F-FEE HDF5 data from the F-DPU Processor. That new process we call the Data Dumper (data_dump). All data that we retrieve from the F-FEE will be sent from the F-DPU Processor to the data dumper using a PUSH-PULL protocol. This ZeroMQ message protocol can push messages to multiple workers, but we will only connect with one worker, i.e. the data dumper.

Our seconds design point above says the F-DPU Processor shall do as few processing as possible and delegate those tasks to other processes. The reason is to make sure the F-DPU Processor can readout the F-FEE data fast enough to avoid any buffer overflow errors. For that reason we put a process in between the F-DPU Processor and the data dumper which will take over the identification of data packets, and processing those packets before they are sent to the data dumper. Communication between the F-DPU Processor and the new data processor is also based on the PUSH-PULL message protocol.

It’s time to put that into a diagram in the hope it will clarify. Don’t worry, we will talk you trough it.

The figure above contains the three processes described above, the F-DPU (FastCameraDPUProcessor), the data processor (FastCameraDataProcessor) and the data dumper (data_dump). The communication between these processes is based on the ZeroMQ PUSH-PULL protocol. Since these processes always run on the CGSE server, we use the faster inter-process communication (IPC) instead of the transmission-control protocol (TCP). We see that data is pushed from the F-DPU to the Data Processor to the data dumper. The F-DPU Processor gets the data from the F-FEE, so there is transport of data and information between the F-DPU and the DSI, which is the SpaceWire interface connected to the F-FEE.

Of course, the data dumper shall also be able to retrieve commands, mainly from the F-DPU, to synchronise data flow or update configuration settings. This is done through the ZeroMQ DEALER-ROUTER protocol which allows to send commands to the data dumper and request a return value only when needed. For example, if the data dumper needs to update its Setup to the latest version, you wouldn’t require a reply when you send a reload-setup command. On the other hand, if you request status information from the data dumper you do expect a reply.

The job of the process manager is to keep track and monitor the execution of running processes during the Camera Tests.

There are known processes that should be running all the time, i.e. the configuration manager, and the Storage Manager. Then, there are device control servers that are dependent on the Site and the Setup at that site. The process manager needs to know which device control servers are available and how to contact them. That information is available in the configuration manager. We need to decide how the interface between the PM and the CM looks like and what information is exchanged in which format.

Synoptics = in a form of a summary or synopsis; taking or involving a comprehensive mental view. According to the Oxford Dictionary.

In all involved test facilities, the EGSE is used to perform the same set of basic operations: monitoring temperatures, changing the intensity of the source and point it somewhere, acquiring images, etc. However, the devices that are used to perform these tasks are not everywhere the same (e.g. the OGSE with its lamp and filterwheels, the DAQs for temperature acquisition, etc.).

Each (device) control server has a dedicated CSV file in which the housekeeping information is stored and often the name of the parameters indicates the test facility at which they were acquired.

It is not inconvenient if user need to memorise the HK names for all test facilities, it also makes the test and analysis scripts more complex.

The Synoptics Manager stores this information in one centralised location (the synoptics file) with generic parameter names.

The tm-dicionary.csv file (further referred to as the "telemetry ™ dictionary") provides an overview of all housekeeping (HK) and metrics parameters in the EGSE system. It is used:

TODO:

The Proxy class automatically tries to connect to its control server. When the control server is up and running, the Proxy loads the device commands from the control server. When the control server is not running, a warning is issued, and the device commands are not loaded. If you then try to call one of the device methods, you will get a NotImplementedError. The ZeroMQ connection initiated by the Proxy will just sit there until the control server comes up, so you can fix the problem with the control server and call the load_commands() method on the Proxy again to load the commands.

The DPU Protocol starts a thread called DPU Processor Thread. This thread is a loop that continuously reads packets from the FEE and writes RMAP commands when there is a command put on the queue. When the control server receives a quit command, this thread also needs to be terminated gracefully since it has a connection and registration with the Storage manager. Therefore, the Protocol class has a quit() method which is called by the control server when it is terminated. The DPU Protocol class implements this quit() method to notify the DPU Processor Thread. Other Protocol implementations usually don’t implement the quit() method.

Another example of a control server that starts a sub-process through its Protocol class in the TCS control server. The control server start the TCS Telemetry process to handle reading housekeeping from the TCS EGSE using a separate port [6667 by default].

The control servers are the single point access to your equipment and devices. Commanding the device is handled by the Protocol and Controller classes on the server side and by the Proxy class on the client side. In addition to commanding your devices, you would also like to have some control on the behaviour of your device control servers. That is where the Service slass comes into play. The Services provide common commands that act on the control server instead of on the device. These commands are send through a Service Proxy that you can request for each of the control servers. Below is a code snippet asking for the service proxy of the hexapod PUNA and requesting the status of the control server process.

Status information for the control servers is also sent out on the monitoring port, so if you need to monitor the process status, its better to subscribe to the monitoring protocol.

There are two commands that you can use to change the frequency by which the control server is sending out information, one is for changing the housekeeping frequency, one is for changing the monitoring frequency.

This function sets the housekeeping frequency (in Hz) to the given freq value. This is only approximate since the frequency is converted into a delay time and the actual execution of the get_housekeeping() function is subject to the load on the server and the overhead of the timing. What will happen is that the delay time that will be applied is never shorter than the average execution time of the get_housekeeping() method increased with a 200ms overhead to allow the control server to respond to other commands.

This function works similar to the set_hk_frequency() but sets the delay time on the get_status() function of the device protocol.

When you want to optimize these frequencies, make sure you have an idea of the execution time of either function. If for instance the get_status() function takes 500ms to complete, it will be impossible to request a monitoring update frequency of 2Hz.

The control.py module defines a number of classes that can be used to send a response on a device command to the client. Responses are handle by the Protocol class in the handle_device_method() method.

The main reason why these classes are provided is because Exceptions that happen in the server process should somehow propagate to the client process. A Response object can contain a return value if the command was executed successfully, or it can contain an Exception object when the device command has failed. If the command has failed or not can be checked by the client with the successful() method.

The Response class has three sub-classes, Success, Failure, and Message. A Success object is returned by the server process with the actual response in the property return_code. In the case of an Exception, the server process logs the error message and packs the Exception object in a Failure object.

Two of the requirements that were formulated when we started the analysis and design of the CGSE was that (1) it shouldn’t be too difficult to implement device controllers and (2) the definition of the commands shall be done at one place only. So we started out with a design where the commands were defined in a YAML file (called the command yaml file) and the code would use this information to build the API for the device. That would make it —from the device developer point of view— rather easy to create the commanding interface for a new device. Unfortunately, the implementation was not that easy or straightforward. We are working in a client-server or even microservices environment which means the device commanding should be available on both the server side (what we call the Controller) and on the client side (what we call the Proxy).

Both the Controller and the Proxy will have to implement the same interface, but we didn’t want to bother the developer with having to implement the interface twice. The Proxy (client) side should be easy, because it doesn’t need any processing or device control, it only needs to forward the command and its arguments to the server to be executed and then wait for the response. On the server it is slightly more complicated since there we have to interact with a device that can be connected by Ethernet, USB, serial,…​ with many different commanding protocols.

We have currently two designs and implementations that handle dynamic commanding. The first implementation we came up with is the @dynamic_interface which is a decorator that is used on the methods in the device interface class. How this works is explained in the section below on Section 6.6.1. The latest implementation is the @dynamic_command which also decorates the interface methods, but does a lot more work with respect to command string generation and response processing. How this work is explained in Section 6.6.2. We have also written up a section on the transition from the old implementation into the new dynamic command implementation. See Section 6.6.3.

An Event is an object that contains all information needed by the listeners in order to process the event notification.

An Event is defined by an event identifier and an event type. Currently, the only event identified is the New Setup event notification with event_id: EVENT_ID.SETUP and event_type: "new_setup". An Event also needs a context which contains information that is needed by the listener in order to handle the event. The context contains event specific information, e.g. in the case of a new Setup, the context also contains the setup identifier (setup_id).

The control server that wants to be notified of an event needs to implement the EventInterface in the Proxy and in the Controller. This interface currently contains only one method, ie. handle_event(event) which takes the event object as its only argument.

Listeners are registered and unregistered through the services interface of the control server. That means the developer of a control server that needs to listen to events doesn’t need to implement the registration functions, only the handling of the events must be implemented.

When a listener is registered, a name and a proxy shall be provided. The name is used as an identifier for the listener, the proxy is the Proxy class for the listening process. The proxy will be used by the producer to notify the listener/consumer.

When registering, another proxy shall be provided, which is the producers proxy, i.e. the proxy for the control server to which the listener wants to register. So, for example, if the storage manager wants to listen to events on new Setups from the configuration manager, that proxy shall be a ConfigurationManagerProxy. The following code snippet should make this clear:

Registrations are issued from the ControlServer and handled by the ServiceProtocol. The registration is executed in a separate thread and retried 5 times with a 10s wait time in between.

Event notifications are handled by executing the handle_event() method on the consumer Proxy. The handle_event command is defined by the EventInterface which is a base class for the Controller and the Proxy. Both the Controller class and the Proxy class need to inherit from the EventInterface.

In this case, we will also need to provide an entry in the command YAML file:

The control server that generates the events keeps a list of registered consumers/listeners. The registration of these listeners is handled by the service protocol. Event notifications are sent out from the ControlServer class, but they usually occur in the Controller class. The Controller therefore shall keep a reference to its control server. When the event is detected, the Controller will use the notify_listeners() method of the ControlServer. This latter method expects an event identifier and the event context. The example code snippet below sends out a new setup loaded notification to all registered listeners.

The notify_listeners() method will execute the actual notification in a separate thread in order not to block the commanding event loop.

The registration, management, and notification of listeners in a control server is handled by the service protocol. That means all control servers automatically have this functionality available. There is only one method that needs to be implemented in the listening control server and that is the handling of the event. That is defined in the EventInterface which the Proxy and the Controller of the listening control server shall inherit and implement.

New commands in Service:

New commands in ControlServer:

The scheduled tasks are functions known in the control server sub-class which will be executed outside the context of commanding and monitoring. The reason we need these scheduled tasks is to handle actions that either take too long during the execution of a command, or when a command request is sent back to the same control server that the original command came from, e.g. request the current Setup. In that latter case we will get a deadlock because both control servers will be waiting for each others reply.

Scheduled tasks are executed from the ControlServers event loop (serve()) after the current command has been finished and the status and housekeeping has been published. Keep in mind that –at this point– the tasks should not take too much time or shall either run in a thread or run asynchronously.

Tasks can also be scheduled after a certain time, e.g. after 10s (approximately), or can be scheduled when a certain condition is true, e.g., when a service comes on-line. The after and when can be combined.

This section explains what the natural flow is for notifications, starting at an event happening in a control server up to processing the event in the registered listener processes.

As an example we will use the event that a new Setup will be loaded in the configuration manager. This can happen when the configuration manager is requested to load a new Setup (load_setup()) or when a client submits a new Setup to the configuration manager (submit_setup()). The configuration manager will then send a notification of this event to its registered listeners. The listeners shall implement a command to accept such an event notification. By convention, this command is handle_event(event) (defined in the EventInterface). So, if you need to be notified about a new Setup, your control server needs to implement the handle_event command in its commanding interface. See further XXXXX

A control server that wants to listen to events needs to register first. This shall be done during startup, usually in the __init__() method of the listening control server. Registration requires a proxy for the notifying control server that will accept the registration, and a listener object (a dictionary) with information about the listener process.

The notifying control server will accept the registration and add the listener to its list of registrations.

When an event occurs at the notifying control server, an event notification will be sent out to all the registered listeners. This is done by the notify_listeners() method of the ControlServer.

The listening control server will receive the notification, execute or schedule a task to act on the event, and acknowledge the proper reception of the notification. This is done in the handle_event() method that needs to be implemented by the listening control server in its Controller class. The handling of the event shall be as fast as possible because the notifying control server will be waiting for an acknowledgment. Therefore, there are currently two options: 1. the task is executed in a thread, 2. the task is scheduled and will be executed from the event loop in the control server (serve()).

The controller now keeps track of the Setup that is currently active. This value changes when a different Setup is loaded (with load_setup()) or when a new Setup is submitted (with submit_setup()) to the configuration manager.

If you want to know which processes have registered to the configuration manager, the status info has been updated for this. The last line now shows the registered listeners:

The Storage Manager can now handle events about a new Setup loaded in the configurations manager and will register to the configuration manager during startup. The Storage Controller now keeps a record of the currently loaded Setup in its Controller.

Since a Setup can only be changed outside the context of an observation, the implementation is not too complicated. The controller has a new method load_setup(id) which is executed as a scheduled task after a notification is received. That task will fetch the Setup from the configuration manager (the reason why we needed scheduled tasks to avoid a deadlock at this point).

The status information from the Storage manager now also includes the ID of the loaded Setup. See second last line in the example output below.

As an example…​ XXXXX

The Dummy module is updated to implement event notification and listener registration. The Dummy control server will register to the configuration manager at start up and will handle any event it receives by logging the event reception and execution. In the shutdown phase after_serve() the Dummy control server unregisters from the configuration control manager.

This section will guide you through the process of implementing event handling into a control server. As an example we will update the Synoptic Manager CS to register to the configuration manager for new Setup events and handle those events in a scheduled task.

So, the summary of changes that we are going to implement is:

The Synoptic Manager is started only after the configuration manager has started up and will therefore be initialised with the proper Setup. No need to add a one-time scheduled task for loading the Setup at startup.

The Setup is defined as a hierarchy of configuration items and their specific settings and definitions. All the hardware and software that make up your complete test setup will be described in the YAML file. As a quick example, the lines below define a typical setup for the Hexapod PUNA that is used at CSL:

The Setup is implemented as a YAML configuration file and loaded into a special Python dictionary. The dictionary is special because it allows you to navigate with a 'dot' notation in addition to keys. The following two lines are equivalent (assuming the Setup is loaded in the variable setup):

Another advantage of this special dictionary is that some fields are interpreted and loaded for you. For example, the device field of the Hexapod starts with class// and provides the class name for the Hexapod device. When you access this field, the class will automatically be instantiate for you and you can start commanding or querying the device. The following example initiates a homing command on the Hexapod controller:

When you want to know which configuration items are defined in the Setup at e.g. the gse level, use the keys() function:

When you want a full printout of the setup.gse, use the print function. This will print out the dictionary structure as loaded from the YAML file.

The Setup will contain configuration information of your test equipment. This includes calibration and conversion information, identification, alignment information, etc. All items that can change from one environment to the next, or from one test to the next, should be fully described in the Setup.

The following list gives an idea of what is contained and managed by a Setup. This list is not comprehensive.

For all of these items the Setup shall hold enough information to uniquely identify the configuration item, but also to reproduce the state of that item, i.e. version numbers of firmware software, transformation matrices for alignment, conversion coefficients, calibration tables, etc.

As described above, the Setup is a special dictionary that contains all the configuration information of your test equipment. This information resides in several files maintained by the configuration control server. You can request a list of Setups that are available from the configuration manager with the list_setups() function. This is a convenience function that will print the list in your Python session.

The above command will contact the configuration control server and request the list of Setups. To load the current active Setup into your session, use the load_setup() method. This function takes one optional argument to load a specific Setup. Now you can work with this setup as explained above, accessing its content and navigate through the hierarchy with the 'dot' notation.

If you need to make a change to one of the configuration items in the Setup, you can just assign a new value to that field. Suppose we have upgraded the firmware of the PUNA Hexapod that is used in this setup, we can use the following commands to make that change:

The change then needs to be submitted to the configuration control server who will put it under configuration control, i.e. assign a new unique Setup identifier and save the entier Setup into a new YAML file.

Whenever you make a change to an item in the Setup, or you add a configuration item, a new Setup will be created with a new unique id as soon as you submit the Setup to the configuration manager.

The configuration control server has no insight or knowledge about the content of a Setup, so you can freely add new items when needed. The simplest way to start with a Setup and adapt it to your current test equipment environment, is to load a Setup that closely resembles your current environment, and only make the changes necessary for the new Setup, then submit the new Setup to the configuration control server.

If you need to start from scratch, create a new empty Setup or feed it with a dictionary that contains already some of the information for the Setup:

If you need to set the firmware version for the Hexapod controller.

This way it is easy to update and maintain your Setup. When ready, submit to the configuration control server as shown above.

If you want to save your Setup temporarily on your local system, use the to_yaml_file() method of Setup. This will save the Setup in your working directory.

From within several places deep in the code of the test scripts, we need access to a certain state of the system and act accordingly. The main state to access is the Setup which provides all configuration and calibration parameters of the test equipment and of the SUT.

Another use is the dry_run state which allows you to execute command sequences (test scripts) without actually sending instructions to the hardware, but just logging that the command would have been executed with its arguments. We could have gone with a singleton pattern, i.e. a class for which only one instance exists in your session, but a singleton is a difficult pattern to test and control. Another possible solution is to use a class for which all instances have a shared state. That means even if the class is instantiated multiple times, its state is the same for all those instances. This pattern is also known as the Borg pattern or the shared-state pattern.

An alternative to provide such functionality is to pass arguments with the required state into all levels until the level/function where the information is needed, even if the argument is never used in most of the intermediate levels.

The name GlobalState is maybe not such a good name as this class actually shares state between its instances, but this shared state is not global in the sense of a global variable. The objects can be instantiated from anywhere at anytime, which is what makes them globally available.

The following sections describe the different states and function provided by the GlobalState. Remember the GlobalState is intended to be used within functions and methods of the test scripts in order to access global state information. That means the GlobalState needs to be initialised for some functions to work properly. Don’t use the GlobalState in Common-EGSE modules. If you need access to the Setup from the CGSE, explicitly request the Setup from the configuration manager using the get_setup() function.

In the egse.system module we have provided a few functions to work with datetime in a consistent way. The most important is the format_datetime() function which is used for instance to create the timestamps for all CSV housekeeping data files.

The format that is returned by default is "YYYY-mm-ddTHH:MM:SS.μs+0000" which means the time is expressed in UTC. The function has a few optional parameters that allows to tune the returned string. This should be used only for specific purposes, use the default always to generate a timestamp for your data.

If you need to parse the timestamp string returned by the format_datetime() function, use the following format string: "%Y-%m-%dT%H:%M:%S.%f%z". We might provide a parse_datetime() function in the future to standardize and simplify the parsing of datetime objects.

If you need a specific date without time, the format_datetime() function takes a string argument that you can use to get the date of today, yesterday, tomorrow and the day before yesterday. This might be useful in scheduling tasks.

The DPU Control Server (used to be called the DPU Simulator in older documents) acts like any other control server. It’s Protocol class (DPUProtocol) starts a DPUController which implements the commands that are defined in the DPUInterface class. Specific DPU commands are sent to the control server using the DPUProxy class. The DPUProxy class also implements the DPUInterface.

The difference with normal device control servers lies in the additional sub-process which handles the communication with the N-FEE. This separate process, the DPUProcessor, is also started by the DPUProtocol and both the DPUProcessor and the DPUController communicate via three multiprocessing Queues, the command queue, the response queue, and a priority command queue. The main task of the DPUProcessor is to communicate with the N-FEE, i.e. command it via SpaceWire RMAP Requests and retrieve housekeeping and data packets that are sent to the Storage manager. The commands that the DPUProcessor must execute are passed through the command queue by the DPUController. Every readout cycle the DPUProcessor checks the command queue and passes any available command to the N-FEE as an RMAP Request.

Commands that are given on the Python prompt are by definition synchronous meaning when you call a function or execute a building block, the function or building block will wait for its return value before finishing. This is usually not a problem, because we most of the time have a pretty good idea how long a calculation or action will take. Most functions return within a few hundred milliseconds or less, which is practically immediate. When commanding the N-FEE however, things are more complicated. The N-FEE has strict timing when it comes to commanding. During the sync period (usually 6.25s unless commanded different) there is a slot of at least 2s at the end of the period, which is reserved for commanding. The sync period starts with a timecode packet followed by the N-FEE housekeeping packet. That takes just a few milliseconds. Depending on the N-FEE mode, we can then expect data packets filling up until 4s after the time code. There can be less or no data packets, leaving more time for commanding. Commanding the N-FEE is done by sending SpaceWire RMAP requests to the N-FEE in that 2s+ timeslot. When we send a command from the Python prompt to the N-FEE, it arrives in the command queue at the DPUProcessor and will be sent to the N-FEE in the next 2s+ command timeslot. When the Command enters the Queue at the beginning of the readout period, i.e. right after the timecode, maximum 4s will pass before the command is actually send to and executed on the N-FEE. All this time, the Python prompt will be blocked while waiting for the response. The next command can only be sent on return of the previous. So, we have two issues to solve, (1) the duration and blocking of all the steps in the commanding chain, and (2) sending more than just one command to the N-FEE in the same timeslot.

XXXXX: add here how we solved this!

The priority queue is also a command queue, but the commands are not sent to the N-FEE. Instead, the commands are executed in the DPUProcessor and are used to either get information about the state of the N-FEE or set/get the internal state of the DPUProcessor. The state of the N-FEE is mirrored and kept up-to-date by the DPUProcessor during the readout cycle. Priority commands are typically to request e.g. the N-FEE mode, or to set an internal DPU parameter. While the command queue is checked only during the allowed RMAP communication period, the priority queue is checked and executed several times during the readout cycle.

The DPU Processor has two ZeroMQ message queues on which it publishes specific information. One message queue is the data distribution queue on which the following information is published. All messages published on this queue are multipart messages with the following IDs to which you can subscribe.

The data distribution message queue is used by processes that need to handle image data like the DPU GUI. There is a second message queue used by the DPU Processor to publish monitoring information. In addition to SYNC_TIMECODE, SYNC_HK_PACKET, NUM_CYCLES, the following information is published on this message queue:

Any process can subscribe to the monitoring queue easily by using the DPUMonitoring class. For example, the following code snippet connects to the DPU Processor monitoring queue, waits until the next timecode is received, then executes some code and waits for the next timecode.

If you need more control over the monitoring and the actions, you can subscribe to the DPU Processor monitoring queue directly. The following code snippet waits for an error flag and raises an alert message on Slack whenever the error flag is not 0:

TBW

Throughout the PLATO camera test campaigns, the CCD numbering, used throughout the CGSE, will remain the same (see figure below), but the underlying numbering, used by MSSL for the N-FEE will be different for EM, PFM, and FM.

TBW

In the lab we use a SpaceWire interface from 4Links. This company provides devices to communicate over SpaceWire, but also to monitor, diagnose, develop your SpaceWire connections.

Two of these devices are used in the different institutes that test the PLATO cameras, i.e. the Diagnostic SpaceWire Interface (DSI) and the portable version of that (P-DSI). These devices have 2, 4, or 8 SpaceWire ports that can work concurrently at a speed up to 400Mbps. A 1Gbps Ethernet port connects your server to the DSI. 4Links provides software to drive the DSI and communicate with the camera over SpaceWire. The main component is a dynamic shared library EtherSpaceLink which is a C library that provides functionality to read and write SpaceWire packets from the active ports.

A Python wrapper module exists and is also delivered by 4Links, but at the time we were developing the CGSE, this Python interface was not available yet. We have implemented our own wrapper functions using the ctypes Python module.

The setup that is used in the lab is shown in the figure below. The N-CAM is connected to SpW port 1 on the DSI and the egse-server is connected to the Ethernet port.

The IP address of the DSI is different for each test house and is therefore configured in the local Settings file. The DSI is listening on TCP port 4949.

The SpaceWireInterface class is the high-level base class that is used to communicate with a SpaceWire device. This class is intended as an interface class and shall not be used directly. We have provided two concrete sub-classes that inherit from the SpaceWireInterface, i.e. the SpaceWireOverDSI and the SpaceWireOverZeroMQ. The interface defines the following methods:

Sub-classes of the interface class can implement additional methods that are specific for that device, but only the interface methods shall be used in processes like the DPU Processor and the FEE Simulator.

The interface class can also be used as a context manager where it will open the connection upon entry and close the connection upon exit or when an exception happens.

Note that the connection is only established by entering the with statement, so, what usually is done is that the SpaceWireInterface sub-class is instantiated at a higher level in a caller function and past into the functions that use it.

TBW

Using the unit tests…​.

TBW

Give a brief overview of the RMAP protocol as it is used in PLATO and describe how RMAP and DSI commands and functions work together.

In this section we will present a more in-depth example of both connections of the SpaceWireInterface, i.e. the sending and the receiving part.

The OGSE simulator by default listens on TCP port 4181 for incoming connections from the OGSE Controller. The controller can be used stand-alone in a Python REPL for testing or can be part of the OGSE Control Server as the last step in the commanding chain.

Start the OGSE simulator as follows:

The OGSE simulator will listen for incoming connections. There can be only one process connected to the OGSE simulator. This behaviour is similar as the hardware device, which only accepts one connection. When a connected process disconnects, the OGSE simulator will accept a new connection.

The OGSE simulator can be killed by pressing CTRL-C in the Terminal where the simulator is running. Alternatively, you can send a TERM or HUP signal to the process:

The OGSE simulator has of course no interlock jack that you can pull out to immediately power off the system. Nevertheless, we can simulate this behaviour by sending a user signal to the ogse_sim process. Sending a USR1 signal will open the interlock when it is closed, and close it when it is open. The USR1 signal works as a toggle for the interlock.

Check the interlock state with the command method get_interlock():

The following commands have been implemented in the simulator:

This class serves as a representation of a nested Python dictionary, enabling navigation through its contents using dot notation, offering an alternative to traditional keyword-based access.

Let’s create a nested dictionary with some configuration information:

Both examples below are equivalent:

and

TBW

TBW

TBW

===

TBW

TBW

TBW

TBW

Depending on your needs, you can choose different methods for the timestamp of your metrics. We will discuss three ways to get and use metrics timestamps:

Option 1. is the best way for timestamps for your metrics when the accuracy of the time when the metric was retrieved is important. The timestamp then itself is a metric (Gauge) and might need to be converted to a float that represents the time in the expected way, depending on the target application.

Option 2. can be used when the device housekeeping doesn’t contain the timestamp of the metrics. This method can be a few to a few tens of milliseconds off depending on the device response time. For some devices and depending on the duration of the query and the number of parameters this can be even longer. One example is the DAQ6510 when tens of temperatures are scanned several times before returning an average value. A full scan can take up to 20s or more.

Option 3. is used when the exact time is not really important and may be a few (tens of) seconds off. This is the case for most temperature measurements evolving slowly.

The time used by Prometheus is the unixtime which means localtime (not UTC) and Epoch 01/01/1970, excluding leap seconds. This is the same as what is returned by the Python time.time() function.

To build up the Task GUI, we distinguish the task button as a function, several of these functions can be grouped in a Python module (a .py file) and all the modules plus additional information needed for the Task GUI is kept in a Python package. The CSL Operator GUI shown above, is located in the package camtest.csl and has the following layout:

Each of these .py files form a group of buttons in the above Task GUI. The __init__.py file is special, it defines camtest.csl as a package, and it defines the command to start the Task GUI [see XXXXX]. The icons folder contains the graphics for the task buttons and the application icon.

Let’s build our own simple Task GUI and start with the most stated and useless function, Hello, World!. We will eventually build a Task GUI with tasks of increasing complexity and guide you through the different steps.

Create a folder yakka[5] that will be our Task GUI package. In the folder create an empty file __init__.py and a file named hello.py.

The hello.py file shall contain the following code:

Make sure you are at the same directory level as the yakka folder and the execute the following command from your terminal. That will start the Task GUI as shown in the screenshot below.

We see the task appearing in the screenshot above. The task text is blue which means it will run immediately when clicked. The tasks name is the name of the function and the task group name is the name of the .py file. The icon is the standard icon used for the task buttons. When you click the task button, the Console shows the following output:

What do we see in this output:

Let’s add another task that takes an argument 'name' as a string with the default value of "John".

In the screenshot above, you can see the effect of the small changes we made in the hello.py. The tasks button group is now called 'Hello Tasks' and the new task we added got the 'Hello…​' name instead of the function name. The new task icon has a different color because it’s is selected. You can also see in the arguments panel that the type hint is picked up and shown in grey and the default name is also filled in grey in the text field. When I put my name there and press the 'Run' button, you can see that the function is called with the proper argument.

TBW

TBW

TBW

TBW

TBW

TBW

Developers: Rik Huygen, Sara Regibo, …​

Instrument Experts: Pierre Royer, Bart Vandenbussche

Do we need more reviewers? The EGSE engineers from INTA and SRON? Somebody from the PCOT?

For each reviewer I will prepare an issue where you can check off the parts which have been reviewed. WHERE TO PUT THE REVIEW REPORTS/COMMENTS?

It is very important that the review is done in a timely fashion. We don’t want to be bothered with this for weeks.

Proposal for reviewer/review items:

Sara Regibo:

Nicolas Beraud:

Rik Huygen:

Pierre Royer:

Bart Vandenbussche:

TODO: Make a checklist!

Be constructive!
Be specific!

The goal is to ship good and maintainable code, it’s not the goal to prove how good or clever we are.

Before the code review, all the code will be run through a number of automated steps:

There will be a specific release for the code review with tag code-review-2020-Q2.

First thing to do is make sure you have access to GitHub and have a GitHub account (personal or professional) so we^[7] can give you access to the relevant repositories. Therefore,

Make sure you have a fork of the repositories that you want to contribute to. Through this fork (which resides on the GitHub server) you will create pull requests. If you do not contribute to any code, you don’t need a fork and can clone the IvS-KULeuven repositories directly.

You can easily create a fork from the IvS-KULeuven repositories by the button in the top-right of the website.

The fork will be created in your personal GitHub account.

Execute the following code in a terminal. It will create a folder where all git repositories reside, then clone your fork of the plato-common-egse. A new folder will be created for plato-common-egse.

You should do the same for the two other repositories, plato-test-scripts and plato-cgse-conf. Remember that you do not need a fork if you are not going to make changes to the repository, like e.g. for the configuration repo.

You should now have three folders in your ~/git location:

Add the following lines to a file that can be sourced into your terminal session, e.g. ~/.bash_profile. Preferably, you should use modules to maintain your environment.

Create a virtual environment for Python 3.8 (the version to be used for the CGSE and TS), then activate the venv and update pip and setuptools packages.

Now install all the packages needed to run the CGSE and TS. These packages are installed from a requirements.txt file which is currently not yet added to the repository. The file can be downloaded from XXXXX.

Also for the test scripts you will need to create a virtual environment. You won’t need to install any additional dependencies since all dependencies are installed in the CGSE virtual environment and this location was added to the PYTHONPATH above.

It’s now time to finally install the Common-EGSE as a developer installation, also called editable mode (-e or --editable). This means the CGSE will be installed as a normal package, only the source code will be linked to the actual checked out source tree in you project. The result is that any changes that you make to the code in your source tree will be used in subsequent executions of scripts or in the Python REPL^[8].

and also do a developer install for plato-test-scripts:

From the project folder, run the invoke task start-core-egse to start up the five core services.

You should now be able to see the core services running on your system:

Now you can also start the process manager GUI with the following command:

If you get the following error of too many open files, increase the limit of open files with ulimit -n 4096 and restart the process manager GUI.

The process manager GUI should now start and show something like the screenshot below depending on the site id you have defined and the Setup ID you have loaded.

It’s important that you keep your local repo up-to-date with the upstream repositories. We have two commands defined for this, update_cgse and update_ts. There are however a few guidelines (or rules) that you have to adhere to in order to keep a clean development environment.

You can request the version of the CGSE and TS as follows [for an explanation on version numbers, see Chapter 2]:

The git commands need to be executed in the project folder or a sub-folder thereof, e.g. ~/git/plato-common-egse.

Related to issue: no issue

The filter wheel control server (fw8smc4_cs) didn’t start when using the process manager GUI (pm_ui). When the button was clicked nothing happened, but we got some error messages in the logger:

There was no clear error message except that the PM was already registered at the storage manager. We thought that could be due to the fact the pm_ui crashed already several times before, or maybe a STORAGE_MNEMONIC in the Settings that was not set correctly. So we checked the Settings YAML file, and also the local settings file. The command to do that is:

We looked at the Setups and inspected the source of the filter wheel control server, but could not relate the problem to anything in the code. Especially, since the control server could be started without problem from the terminal.

So we started to look into all kinds of settings that were important when starting a control server from the process manager, e.g. the DEVICE_SETTINGS variable that is expected in the module and should contain the definition of the ControlServer. XXXXX: xref to document/section explaining this. We fixed a number of these things, see for example PR #1963, but still the problem remained.

We were still confused about the message that the PM was already registered at the Storage Manager. Why does this appear here? Why would the process manager try to register again. Looking at the debug messages before the error, we saw that the process manager actually restarted. That’s of course the reason why it tries to register to the Storage, and the fact that it is already registered is probably due to the fact that the process manager crashed. So, instructing the pm_cs to start the fw8smc4_cs (by clicking the button in the pm_ui) crashes the process manager. And it is of course immediately restarted by the Systemd services.

So, we checked the logging messages of the Systemd for the pm_cs.services:

The problem seems to be that the pm_cs can not load the libximc shared library which is needed for the filter wheel control server. This works in the terminal, because there the LD_LIBRARY_PATH is set, but it is not known to the process manager control server. To fix this, the environment variable must be set in the /cgse/env.txt file that is used in the services file to start the pm_cs.services. The content of the files:

PYTHONPATH PLATO_CONF_DATA_LOCATION PLATO_CONF_REPO_LOCATION PLATO_DATA_STORAGE_LOCATION PLATO_LOG_FILE_LOCATION LD_LIBRARY_PATH

and

After this fix, the fw8smc4_cs could be started from the process manager GUI.

So, I let myself work on a side track today and yesterday. I was doing some monitoring tests with the dummy control server and noticed that the Dummy CS would not properly terminate. It was hanging and it was not clear why.

I went through the whole chain of calls to close() functions on all sockets to find out if there was one still missing and if maybe I should use the linger=0 option etc. I placed log messages before and after all these actions, used different methods to stop and start sub processes, …​ It took me way too long to find out…​

When I put logging statements around the term() call for the ZeroMQ context of the control server, I noticed these message never appeared. Thinking therefore the problem I was trying to solve would appear before these statements was again wrong. The reason they did not appear was that I close down all ZeroMQ logging handlers before I terminate the context, the logging level for the standard logger was set to INFO instead of DEBUG.

So, I will have to use logging.info there instead.

Since the control server was hanging when terminating the ZeroMQ context, checking if some port number was still open sounded like a good idea. I found that a connection to the configuration manager commanding port (6000) was still open for the dummy_cs process (PID 36619 at that time):

So, somewhere there must be a connection to the configuration control server that was not closed properly. I started searching the code and finally came to this piece of art (🙄) in the control.py module.

This is in the unregister_as_listener() method of the ControlServer class and the idea behind this code was to quickly check if the CM CS was active before sending a command to it that takes more time (because we use 5 retries). Turns out to be a bad idea for several reasons.

So, why not directly using the Proxy with a short timeout. The Proxy does a ping() when entering the context. So, I came up with this snippet for now:

Now, the next question is if we do need this at all. In the register_as_listener() method, we don’t first check if the control server is active, so why did I think it is necessary here? There are some comments in the code that try to clarify this, but I will spill the beans, the unregister…​ method is usually called when shutting down the services and it might be that the configuration manager is already shut down at the time we want to unregister.

This section describes the main components of the Common-EGSE system.

The Common-EGSE provides functionality to configure, control and/or readout the following devices:

All data that is output by any of these devices will be archived in global and/or test specific files. The file types and formats are described in section Error! Reference source not found. Error! Reference source not found..

The Common-EGSE consists of several components, each with specific responsibilities, that communicate over a ZeroMQ network protocol. The following components have been identified and are part of the core functionality of the Common-EGSE:

Figure 1 above summarises the main components in the core Common-EGSE and test setup and defines their connections.