Guide to config-files¶

General remarks¶

Instead of .csv files als in previos versions, now .toml markup is used. An overview is provided here https://github.com/toml-lang/toml

Setup sample files¶

To illustrate the configuration, some sample files are needed.

# make a folder in the the directory as larda, larda-cfg, and larda-connectordump
mkdir -p example-data/categorize/2018 && cd example-data/categorize/2018/
wget http://devcloudnet.fmi.fi/cnet/limassol/processed/categorize/2018/20180208_limassol_categorize.nc
cd ../../../
mkdir -p example-data/classification/2018 && example-data/classification/2018
wget http://devcloudnet.fmi.fi/cnet/limassol/products/classification/2018/20180208_limassol_classification.nc

Campaigns config¶

The larda-cfg/campaigns.toml config file is used to provide the general context of the data.

[lacros_cycare_example]
    location = "Limassol"
    coordinates = [34.677, 33.038]
    altitude = 11
    mira_azi_zero = 154
    duration = [["20161018", "today"]]
    systems = ["CLOUDNET", "MIRA"]
    system_only.CLOUDNET =  [['20161101', '20190107'], ['20180101', '20180214']]
    system_only.MIRA =  [['20161101', '20180401']]
    cloudnet_stationname = 'limassol'
    info_text_loc = 'default'
    #info_text_loc = 'info_lacros.toml'
    param_config_file = 'params_cycare_example.toml'
    connectordump = '/home/larda3/larda-connectordump/'

Parameter config¶

The second level of configuration defines the parameters for each system in a file such as params_cycare_example.toml. The file lists different systems, such as MIRA, CLOUDNET or POLLYNET. Each systems’ config has three parts. An example is given below.

path¶

Defines the paths, where to find the (netcdf) files containing the parameters.

base_dir: directory to start filesearch
matching_subdirs: regex, that describes the subpaths (including filename) that are matching_subdirs
date_in_filename: named groups in regex that identify the part of the filename, that contains the date

Note

implicitly it is assumed, that the timestamp in the filename is the beginning of measurements

generic¶

Set of properties, that is used for each parameter as default if not specified explicitly for the parameter itself.

params¶

Define the settings for each variable, that should be handled by larda.

time_variable: name of the time variable in the netcdf file
time_conversion: function that converts the time variable to unix timestamp
time_microsec_variable: if given specifies the variable containing the microseconds
time_millisec_variable: if given specifies the variable containing the milliseconds
range_variable: name of the range variable in the netcdf file
range_conversion: function that converts the range variable to meters. 'none' if already given in that unit
var_conversion: function that converts the variable variable to meters. 'none' if no conversion is desired
colormap: colormap to use by default
which_path: name of the path definition that matches the files for this parameter
ncreader: which reader to use
identifier_rg_unit: name of the range unit attribute in the netcdf varibale
identifier_var_unit: name of the var unit attribute in the netcdf varibale
identifier_var_lims: name of the var limits attribute in the netcdf varibale
var_lims: define limits of variable directly
var_name: name of the variable
vel_variable: velocity variable for reading spectra
dimorder: toggle the order of dimensions (i.e. mira nc file)
meta.*: dictionary of meta information extracted from variables, var attributes or global attributes

Example¶

The section for the Cloudnet configration in the params_cycare_example.toml might look like below. The absolute paths in base_dir will likely have to be adapted.

[CLOUDNET]
    [CLOUDNET.path.categorize]
        # mastering regex (here to exclude ppi and stuff)
        base_dir = '/home/larda3/example-data/categorize/'
        matching_subdirs = '(\d{4}\/\d{8}.*.nc)'
        date_in_filename = '(?P<year>\d{4})(?P<month>\d{2})(?P<day>\d{2})_'
    [CLOUDNET.path.productsclass]
        # mastering regex (here to exclude ppi and stuff)
        base_dir = '/home/larda3/example-data/classification/'
        matching_subdirs = '(\d{4}\/\d{8}.*.nc)'
        date_in_filename = '(?P<year>\d{4})(?P<month>\d{2})(?P<day>\d{2})_'
    [CLOUDNET.generic]
        # this general settings need to be handed down to the params
        time_variable = 'time'
        range_variable = 'height'
        colormap = "gist_rainbow"
        time_conversion = 'beginofday'
        range_conversion = 'sealevel2range'
        var_conversion = 'none'
        ncreader = 'timeheight'
        # if identifier is given read from ncfile, else define here
        identifier_rg_unit = 'units'
        identifier_var_unit = 'units'
        identifier_var_lims = 'plot_range'
        identifier_fill_value = 'missing_value'
        #var_lims = [-40, 20]
        meta.version = "gattr.software_version"
        meta.history = "gattr.history"
        meta.source = "vattr.source"
        meta.latitude = "var.latitude"
    [CLOUDNET.params.Z]
        variable_name = 'Z'
        which_path = 'categorize'
        var_conversion = 'z2lin'
    [CLOUDNET.params.LDR]
        variable_name = 'ldr'
        which_path = 'categorize'
        var_conversion = 'z2lin'
    [CLOUDNET.params.T]
        variable_name = 'temperature'
        which_path = 'categorize'
        range_variable = 'model_height'
    [CLOUDNET.params.beta]
        variable_name = 'beta'
        which_path = 'categorize'
    [CLOUDNET.params.depol]
        variable_name = 'lidar_depolarisation'
        which_path = 'categorize'
        var_unit = '%'
        var_lims = [0.0, 0.3]
    [CLOUDNET.params.CLASS]
        variable_name = 'target_classification'
        which_path = 'productsclass'
        var_unit = ""
        var_lims = [0, 10]
        colormap = 'cloudnet_target'
        fill_value = -99

Note

var_conversion allows for chained functions, such as var_conversion = 'z2lin,extrfromaxis2(0)'. See pyLARDA.helpers.get_converter_array().

A template option is available for repeating datasets in different campaigns:

[CLOUDNET]
    template = 'temp_cloudnet.toml'
    [CLOUDNET.path.categorize]
        # mastering regex (here to exclude ppi and stuff)
        base_dir = '/home/larda3/example-data/categorize/'
        matching_subdirs = '(\d{4}\/\d{8}.*.nc)'
        date_in_filename = '(?P<year>\d{4})(?P<month>\d{2})(?P<day>\d{2})_'
    [CLOUDNET.path.productsclass]
        # mastering regex (here to exclude ppi and stuff)
        base_dir = '/home/larda3/example-data/classification/'
        matching_subdirs = '(\d{4}\/\d{8}.*.nc)'
        date_in_filename = '(?P<year>\d{4})(?P<month>\d{2})(?P<day>\d{2})_'

The generic and params section are then defined in larda-cfg/temp_cloudnet.toml. In the pyLARDA.ParameterInfo.ParameterInfo, the template is updated with the campaign configuration. Hence, single generic or params configurations in the template can be overwritten.

The configuration can be checked by running python3 ListCollector.py Afterwards the connectordump at larda-connectordump/lacros_cycare_example/connector_CLOUDNET.json should look similar to

{
"categorize": [
    [
    [
        "20180208-000000",
        "20180209-000000"
    ],
    "./2018/20180208_limassol_categorize.nc"
    ]
],
"productsclass": [
    [
    [
        "20180208-000000",
        "20180209-000000"
    ],
    "./2018/20180208_limassol_classification.nc"
    ]
]
}