Skip to content

AGATA Workflow

This document provides a guide to help the users understanding the global AGATA worklflow. It includes a global description of the NARVAL framework, the data structuration for replays, and a description of the femul emulator used for data reprocessing.

People involved in this document:

D. Bazzacco, A. Boston, E. Clement, N. Dosmes, J. Dudouet, A. Gadea, F. Holloway, L. Hongjie, A. Korichi, N. Lalovic, E. Legay, J. Ljungvall, C. Michelagnoli, R. Perez, D. Ralet, M. Siciliano and O. Stezowski

Some general statements#

NARVAL, actors and the AGAPRO package#

The AGATA DAQ box is based on NARVAL [see also the section 'the ADF component' in the GammaWare User's Guide]. As a consequence of such a framework, the AGATA data is processed through consecutive calls of ACTORS, each actor being in charge to delivering useful data for the next one in the chain. Even if NARVAL is itself written in ADA, it can bind other languages (C, C++, ...). However, the choice of the AGATA collaboration has been to develop algorithms in the C++ language. The most relevant actors are part of a package names AGAPRO [see How to install the AGAPRO package in the the installation section].

The AGATA local level Processing#

At the local level processing, three actors, also called filters in NARVAL's terminology, process the data flow. Those actors, Preprocessing, PSA (Pulse Shape Analysis) and PostPSA1 are in charge respectively to prepare traces (energy calibration / time alignments), to determine hits from traces and to apply additional corrections/filters to prepare the data to be merged. At this stage one achieves the global level processing which is mainly based on the gamma-ray tracking. Such a chain is illustrated in the following picture:

image

Before any processing, an actor is initialized first using a configuration file which has in principle the .conf extension. Such a file contains all the parameters required by the algorithm to work properly. This is illustrated in the following picture:

image

Note

To manage all the configuration files needed by all the actors, a python script, called gen_conf.py , is provided.

In addition, to determine the best parameters for a given data set, specific works (or actions) are required which are based on spectra and/or on more complex inputs (traces, events, energy) at different levels in the processing of the data flow. Several mechanisms are available to do this. Traces/events can be read from the data stream (in the ADF format) out of any actors and thus can be used to build spectra. This is the way the Watchers work. For historical reasons, spectra, traces/events can be produced also directly by the actor itself2 and saved on disk together with the ADF files.

The spectra produced by the actors are saved in a specific format that can be read by TkT or by the GammaWare package. Specific treatments are then respectively realized through dedicated command line programs, available in the AGAPRO package, or through graphical user's interfaces in the GW/Watchers.

Replay of Data / Directories organization#

The data produced by an experiment are stored in a single (unix) directory itself composed of several sub-directories one per run. For each run, one can find the data produced (Data sub-directory) at different level in the data flow as well as the files used to configure (Conf sub-directory) the different actors.

/agatadisks/eXXX/eXXX/run_0001.dat.14-04-17_11h12m53s
|-- Conf
|-- Data
|-- gen_conf.py

The two sub-directories are themselves divided in sub-directories, one per Germanium crystal for the local level processing and several ones for the ancillaries and the global level processing:

/agatadisks/eXXX/eXXX/run_0001.dat.14-04-17_11h12m53s
| -- Conf
| | -- 00A
| | -- 00B
| | -- ...
| | -- 14C
| | -- Builder
| | -- Global
| | -- Merger
| -- Data 
| | -- 00A
| | -- 00B
| | -- ...
| | -- 14C
| | -- Builder
| | -- Global
| | -- Merger

Here is also a snapshot of a typical content of the bottom directories:

| -- Conf
| | -- 14C
| | | -- BasicAFC.conf
| | | -- BasicAFP.conf
| | | -- CrystalProducer.conf
| | | -- CrystalProducerATCA.conf
| | | -- PSAFilter.conf
| | | -- PostPSAFilter.conf
| | | -- PreprocessingFilter.conf
| | | -- PreprocessingFilterPSA.conf
| | -- Builder
| | | -- BasicAFC.conf
| | | -- CrystalPositionLookUpTable
| | | -- EventBuilder.conf
| | | -- TrackingFilter.conf
| | -- Merger
| | | -- BasicAFC.conf
| | | -- CrystalPositionLookUpTable
| | | -- EventMerger.conf
| | | -- TrackingFilter.conf

Replay of Data/Emulators#

In the process to optimize the parameters to get the best quality as possible for your data, it is required to apply one or several times the same algorithm, or actor, using different configuration files. This could be done using NARVAL itself, on site or out site. Another possibility is to use what is called emulators i.e. other frameworks in charge of organizing / running the processing of data by many different actors.

Narval (and Narval standalone)#

How to replay with NARVAL on site / out site ... To be written

Femul#

FEMUL (Flat EMULator) is able to run complex topologies including DISPATCHER i.e. actors having several input lines and one output line. There are two such dispatchers, EventBuilder and EventMerger: the former being in charge of building AGATA events (crystals in coincidence from the PSA_XXXX.adf and resulting in Builder_XXXX.adf files) and the later allows one to build coincidences with ancillaries (resulting in merged_XXXX.adf files). All actors are part of the AGAPRO package. The procedure to get/compile/install this program is given in the Cookbook [see How to install the femul emulator in the Cookbook].

The topology (list of detectors and actors to apply on the data flow) is built using an ascii Topology.conf file and the way to run is3:

femul Topology.conf

Depending on the task you would like to perform, you may need to adapt the topology given in Topology.conf. The most commonly actors used are:

  • Producer actors (to start a NARVAL chain):

    • CrystalProducerATCA \(\rightarrow\) To start a replay from traces (*.cdat) files

    • BasicAFP \(\rightarrow\) To start a replay from ADF files (Basic Agata Frame Producer)

  • Filter actors (to apply calibrations/algorithms on the data flow):

    • PreprocessingFilterPSA \(\rightarrow\) To apply energy and time calibrations/corrections

    • PSAFilter \(\rightarrow\) To execute the Pulse Shape Analysis algorithm

    • PostPSAFilter \(\rightarrow\) For final corrections after PSA (neutron damage, re-calibrations...)

    • TrackingFilterOFT \(\rightarrow\) To process the tracking algorithm

  • Dispatcher actors (to merge several NARVAL chains in one):

    • EventBuilder \(\rightarrow\) To build a AGATA event within a specific timing window

    • EventMerger \(\rightarrow\) To build a global event between AGATA and an ancillary detector in a specific timing window

  • Consumer actors (to end a NARVAL chain):

    • BasicAFC \(\rightarrow\) To end a replay and write ADF files on disk (Basic Agata Frame Consumer)

    • None \(\rightarrow\) To end a replay without writing the output files (used when only spectra files are used in calibration procedures)

Here are three different examples of such topology file:

The first one shows how to read traces (fevent_mezzdata.cdat.*) from many crystals, run up to the PSA and then dump hits in ADF files. (eg: PSA_XXXX.adf):

LOOP CRY 00B 00C 01B 01C 04B 04C 05A 05B 05C 06A 06B 06C

Chain 4   CRY
Producer  CrystalProducerATCA
Filter    PreprocessingFilterPSA
Filter    PSAFilter
Consumer  BasicAFC
ENDLOOP

The second example shows how to read psa hits (psa*.adf) from many crystals, perform some post PSA task, build AGATA events, apply tracking and then dump tracked gamma-rays in ADF files (Tracked*.adf):

LOOP CRY 00B 00C 01B 01C 04B 04C 05A 05B 05C 06A 06B 06C

Chain 3     CRY
Producer    BasicAFP 
Filter      PostPSAFilter
Dispatcher  EventBuilder
ENDLOOP

Chain 3     Global/
Builder     EventBuilder
Filter      TrackingFilter
Consumer    BasicAFC

The last example shows how to read psa hits (psa*.adf) from many crystals, perform post PSA, build AGATA events, merge them with some ancillary data before applying tracking and then dump tracked gamma-rays in an ADF file:

LOOP CRY 00B 00C 01B 01C 04B 04C 05A 05B 05C 06A 06B 06C

Chain 3       CRY 
Producer      BasicAFP
Filter        PostPSAFilter
Dispatcher    EventBuilder
ENDLOOP

Chain 3       Builder/
Builder       EventBuilder
Consumer      BasicAFC
Dispatcher    EventMerger

Chain 2       ancillary/
Producer      BasicAFP
Dispatcher    EventMerger

Chain 3       Merger/
Builder       EventMerger
Filter        TrackingFilterOFT
Consumer      BasicAFC

To replay data, one should create a top destination directory containing the same structure as the one produced online. The Conf sub-directory and the gen_conf.py should be copied from the initial one, and the original Data should be linked to avoid copies of large amount of data. The gen_conf.py is then used to adapt the configuration files to your current directory. It will also produce an Out sub-directory where the files produced by the replay will be stored. Here is an example to prepare a replay folder for run_xxxx, where the raw data are located in /agatadisks/:

mkdir Replay
cd Replay
mkdir run_xxxx
cd run_xxxx
cp -r /agatadisks/run_xxxx/Conf ./
cp /agatadisks/run_xxxx/gen_conf.py ./
(Here edit the gen_conf.py as explain in the following)
ln -s /agatadisks/run_xxxx/Data Data
python gen_conf.py

Note

The command: python "gen_conf.py -h" show a help When copying the gen_conf.py , take care of changing ONLINE into OFFLINE and NARVAL into femul.

Here is the beginning of the gen_conf.py script showing some parameters to be changed for the replay you would like to perform.

"""
Script to generate the configuration files for the replay of AGATA data using the
multi-process distributed system Narval or the single-process emulator femul.
The script is divided in a few sections, some of which are specific to the actual
analysis (and therefore are likely to be changed by the user) while some others are
normally not to be touched:
0) Type of analysis and replacement macros (in the style of the shell) used to parametrize
the commands listed in 2).
1) The structure of the actual analysis is defined by the variables PROGTYPE and CONFTYPE and
by the dictionaries Topology and Actors. The dictionary ExtraFiles contains a list of files,
which are needed by the analysis but are not generated by this script (e.g. calibrations,
mappings, ...); these files can be copied from a previous analysis if the script is started
with the option -o or --old (python gen_conf.py -h to get the list of accepted options)
2) The command lines to be written in the conf files of the actors defined in Actors{}.
Uncomment/comment/modify the command lines and their parameters
3) A small database defining the position and the PSA signal basis of the germanium crystals.
This part should not need to be changed.  

To get a list of command line arguments, launch the script as:   gen_conf.py -h
To get a list of keywords accepted by the vaious actors:         femul -k
"""
###############################################################################################
###################  0  Type of analysis and replacement symbols  #############################
###############################################################################################

PROGTYPE='femul'    # NARVAL or femul   (to choose between os.getcwd() and '' for CWD)
CONFTYPE='OFFLINE'  # ONLINE or OFFLINE (used just to exclude the ReadDataDir line in the Producers)

MACROS={            # various replacements for symbols defined in 2).
'$CONFDIR'      : 'Conf',                      # this will be prefixed by CWD/
'$READDIR'      : 'Data',                      # this will be prefixed by CWD/; if ONLINE this will not be written
'$SAVEDIR'      : 'Out',                       # this will be prefixed by CWD/; if ONLINE this will be replaced by $READDIR
'$ANCILLARY'    : 'Ancillary',                 # this will be prefixed by CWD/
'$GLOBAL'       : 'Global',                    # this will be prefixed by CWD/
'$PSABASE'      : '/agatadisks/bases/ADL',     # standard place at LNL/Linux
'$CRYSTAL_ID'   : "",                          # the actual value is defined in GeDataBase
'$SIGNAL_BASIS' : "",                          # the actual value is defined in GeDataBase
'$CRYSTAL'      : "",                          # the actual value taken from Topology['CRYSTAL']
}

###############################################################################################
###################  1  Structure of analysis  ################################################
###############################################################################################

Topology={    # The directories to be generated in Conf, Data and Out
'CRYSTAL'   : "00A 00B 00C 01A 01B 01C 02A 02B 02C 03A 03B 03C 04A 04B 04C 05A 05B 05C",
'ANCILLARY' : "Ancillary",
'BUILDER'   : "Builder",
'MERGER'    : "Merger",
}

# The name of the used actors must correspond to one of the tuples defined in the following section.
# This requirement creates a problem for BasicAFP and BasicAFC when they are used in chains of different type
# (e.g. after PSA and after Tracking) and one wants to define chain-specific names for their input/output files.
# The solution is to suffix the name of the chain-type (e.g. _CRYSTAL or _GLOBAL or any other), to the defining tuple. 
# This suffix will be silently removed from the actual name of the generated configuration files.

Actors={      # These are the xxxx.conf files to be generated 
'CRYSTAL'   : "CrystalProducer BasicAFP_CRYSTAL PreprocessingFilter PSAFilter PostPSAFilter BasicAFC_CRYSTAL",
'ANCILLARY' : "BasicAFP_ANCYLLARY AncillaryFilter BasicAFC_ANCYLLARY",
'BUILDER'   : "BasicAFP_BUILDER EventBuilder BasicAFC_BUILDER",
'MERGER'    : "EventMerger TrackingFilter BasicAFC_MERGER",
}

ExtraFiles={  # If not already present, these files can be copied from a directory specified in the command line
'CRYSTAL'   : "CrystalProducerATCA.conf PreprocessingFilterPSA.conf xinv_1325-1340.cal xdir_1325-1340.cal",
'ANCILLARY' : "AncillaryFilter.conf",
'BUILDER'   : "CrystalPositionLookUpTable",
'MERGER'    : "CrystalPositionLookUpTable",
}

###############################################################################################
###################  2  Tuples specifying the content of the configuration files  #############
###############################################################################################

In case of replay of the PSA actor, the ADL bases need to be downloaded from the AGATA DAQ box or from Cologne web-site. Contact us for obtaining the password (contacts on the home page). The path to the ADL bases needs to be updated in the gen_conf.py script (in the $PSABASE variable).

The CrystalProducer actor needs also to be modified to specify to use the traces files as input. For this, uncomment the lines "InputDataFile" and "AllInputFiles":

#########################
CrystalProducer=(
"ActualClass           CrystalProducerATCA",    
"CrystalID             $CRYSTAL_ID",            
"ReadDataDir           $READDIR/$CRYSTAL",      
"SaveDataDir           $SAVEDIR/$CRYSTAL",        
"TraceLength           100",               
"WriteDataMask         0",
"InputDataFile         event_mezzdata.cdat", 
"AllInputFiles",
)

Finally, make sure that the option "NoMultiHist" is commented in the different actors of the gen_conf.py during the calibration procedure. This allow to produce the spectra files used in the different steps of this document.

  1. Depending on the data flow processed, this filter may not be mandatory ↩

  2. Depending on the configuration of the processing chain at compilation and from the different .conf files ↩

  3. femul -h to get some help from the command line ↩