API documentation

pyLARDA main module

class pyLARDA.LARDA(data_source='local', uri=None)

init a new larda instance

Parameters:

• data_source (str, optional) – either 'local', 'remote' or 'filepath'

• uri – link to backend

Returns:

larda object

connect(*args, **kwargs)

switch to decide whether connect to local or remote data source

connect_local(camp_name, build_lists=True, filt=None)

built the connector list for the specified campaign (only valid systems are considered) the connectors are instances of the Connector.Connector Class

NEW: one connector per system then the params are parts of this system

Parameters:

• camp_name (str) – name of campaign as defined in campaigns.toml

• build_lists (Bool, optional) – Flag to build the filelists or not (with many files this may take some time)

• filt (list, optional) – Filter for name ['system', 'MIRA'] or insturment identifiert ['instr_name', 'hatpro_g2_lacros']

connect_remote(camp_name, **kwargs)

connect to a remote data source

Parameters:

camp_name – name of the campaign in the remote source

connect_templates(c_info={})

set up the connector for the filepath option

Parameters:

c_info (optional) – campaign meta info

read(system, parameter, time_interval, *further_slices, **kwargs)

Parameters:

• system (str) – identifier for the system

• parameter (str) – choosen param

• time_interval – [dt, dt] time interval, or [dt] one time

• *further_slices – range, vel,.. [0, max] or [3000]

Returns:

the dictionary with data

description(system, parameter)

Parameters:

• system (str) – identifier for the system

• parameter (str) – choosen param

Returns:

the description tag as string

get_avail_dict(*args)

get the no files for each date :param *args: either empty or system, parameter

Returns:

the number of files per day for each system

pyLARDA.resolve_today(lst)

resolve ‘today’ in [[‘2012101’, ‘today’]]

class pyLARDA.LARDA_campaign(config_dir, campaign_file)

provides information about campaigns collected in LARDA

get_campaign_list()

list of all campaign names stored in csv-file

assign_campaign(name)

dedicate object to a specific campaign

class pyLARDA.LARDA_campaign_remote(info_dict)

store the campaign information in a similar structure as for local

pyLARDA.Connector

pyLARDA.Connector.convert_regex_date_to_dt(re_date)

convert a re_date dict to datetime

Warning

When using 2 digit years (i.e. RPG) a 20 will be added in front

Parameters:

re_date (dict) – result of the regex search with keys

Returns:

datetime

pyLARDA.Connector.convert_to_datestring(datepattern, f)

convert the date in a (file-)string to dt

Parameters:

• datepatttern – a python regex definition with named groups

• f – the string

Returns:

datetime

pyLARDA.Connector.setupreader(paraminfo) → Callable

obtain the reader from the paraminfo

pyLARDA.Connector.setup_valid_date_filter(valid_dates) → Callable

validator function for chunks of valid dates

Parameters:

valid_dates – list of [begin, end] in ‘YYYYMMDD’

Returns:

a single argument (‘YYYYMMDD-HHMMSS’) validator function

pyLARDA.Connector.path_walk(top, prefilter='.*', topdown=False, followlinks=False)

pendant for os.walk

pyLARDA.Connector.guess_end(dates)

estimate the end of a file

Returns:

list of pairs [begin, end]

class pyLARDA.Connector.Connector_remote(camp_name, system, plain_dict, uri)

connect the data (from the a remote source) to larda

Parameters:

• camp_name (str) – campaign name

• system (str) – system identifier

• plain_dict (dict) – connector meta info

• uri (str) – address of the remote source

collect(param, time_interval, *further_intervals, **kwargs) → dict

collect the data from a parameter for the given intervals

Parameters:

• param (str)

• time_interval – list of begin and end datetime

• *further_intervals – range, velocity, …

• **interp_rg_join – interpolate range during join

Returns:

data_container

description(param)

get the description str

get_as_plain_dict() → dict

put the most important information of the connector into a plain dict (for http tranfer)

pyLARDA.Connector.walk_str(pathinfo)

match the names and subdirs with regex using string

only works for unix systems, but is reasonably fast

Parameters:

pathinfo – dict

Returns:

all_files

pyLARDA.Connector.walk_pathlib(pathinfo)

match the names and subdirs with regex using pathlib

should give cross-platform compatability, but comes with a performance penalty (>3x)

Parameters:

pathinfo – dict

Returns:

all_files

class pyLARDA.Connector.Connector(system, system_info, valid_dates, description_dir=None)

connect the data (from the ncfiles/local sources) to larda

Parameters:

• system (str) – system identifier

• system_info (dict) – dict info loaded from toml

• valid_dates (list of lists) – list of begin and end datetime

• description_dir (optional) – dir with the description rst

build_filehandler()

scrape the directories and build the filehandler

save_filehandler(path, camp_name)

save the filehandler to json file

load_filehandler(path, camp_name)

load the filehandler from the json file

collect(param, time_interval, *further_intervals, **kwargs) → dict

collect the data from a parameter for the given intervals

Parameters:

• param (str)

• time_interval – list of begin and end datetime

• *further_intervals – range, velocity, …

• **interp_rg_join – interpolate range during join

Returns:

data_container

collect_path(param, time_interval, *further_intervals, **kwargs) → dict

“

Returns:

data_container

get_as_plain_dict() → dict

put the most important information of the connector into a plain dict (for http tranfer)

Returns:

connector information

{params: {param_name: fileidentifier, ...},
avail: {fileidentifier: {"YYYYMMDD": no_files, ...}, ...}

files_per_day(which_path) → dict

replaces days_available and day_available

Returns:

dict with days and no of files

{'YYYYMMDD': no of files, ...}

pyLARDA.helpers

pyLARDA.helpers.get_converter_array(string, **kwargs)

colletion of converters that works on arrays combines time, range and varconverters (i see no conceptual separation here)

the maskconverter becomes relevant, if the order is no time, range, whatever (as in mira spec)

chaining example: `var_conversion = 'z2lin,extrfromaxis2(0)'`

Returns:

(varconverter, maskconverter) which both are functions

pyLARDA.helpers.flatten(xs)

flatten inhomogeneous deep lists e.g. [[1,2,3],4,5,[6,[7,8],9],10]

pyLARDA.helpers.since2001_to_dt(s)

seconds since 2001-01-01 to datetime

pyLARDA.helpers.dt_to_ts(dt)

datetime to unix timestamp

pyLARDA.helpers.ts_to_dt(ts)

unix timestamp to dt

pyLARDA.helpers.argnearest(array, value)

find the index of the nearest value in a sorted array for example time or range axis

Parameters:

• array (np.array) – sorted array with values, list will be converted to 1D array

• value – value to find

Returns:

index

pyLARDA.helpers.nearest(array, pivot)

find the nearest value to a given one

Parameters:

• array (np.array) – sorted array with values

• pivot – value to find

Returns:

value with smallest distance

pyLARDA.helpers.lin2z(array)

linear values to dB (for np.array or single number)

pyLARDA.helpers.z2lin(array)

dB to linear values (for np.array or single number)

pyLARDA.helpers.raw2Z(array, **kwargs)

raw signal units (MRR-Pro) to reflectivity Z

pyLARDA.helpers.get_extrfromaxis0(string)

get function that extracts given index from axis2

pyLARDA.helpers.get_extrfromaxis1(string)

get function that extracts given index from axis2

pyLARDA.helpers.get_extrfromaxis2(string)

get function that extracts given index from axis2

pyLARDA.helpers.fill_with(array, mask, fill)

fill an array where mask is true with fill value

pyLARDA.helpers.guess_str_to_dict(string)

try to convert a text string into a dict intended to be used in the var_def

Returns:

dict with flag as key and description string

pyLARDA.helpers.reshape_spectra(data)

This function reshapes time, range and var variables of a data container and returns numpy arrays.

Parameters:

data (dict) – data container

Returns:

list with

• ts (numpy.array): time stamp numpy array, dim = (n_time,)

• rg (numpy.array): range stamp numpy array, dim = (n_range,)

• var (numpy.array): values of the spectra numpy array, dim = (n_time, n_range, n_vel)

pyLARDA.helpers.pformat(data, verbose=False)

return a pretty string from a data_container

pyLARDA.helpers.isKthBitSet(n, k)

Function to check if a certain bit of a number is set (required to analyse quality flags)

pyLARDA.helpers.pprint(data, verbose=False)

print a pretty representation of the data container

pyLARDA.helpers.extract_case_from_excel_sheet(data_loc, sheet_nr=0, **kwargs)

This function extracts information from an excel sheet. It can be used for different scenarios. The first row of the excel sheet contains the headline, defined as follows:

	A	B	C	D	E	F	G	H
1	date	start	end	h0	hend	MDF	noise_fac	notes

OR

	A	B	C	D	E	F
1	datestart	dateend	h0	hend	noise_fac	notes

The following rows contain the cases of interest. Make sure that the ALL the data in the excel sheet is formatted as string! The data has to be provided in the following syntax:

• date (string): format YYYYMMDD

• start (string): format HHMMSS

• datestart (string): format YYYYMMDDHHMMSS

• dateend (string): format YYYYMMDDHHMMSS

• end (string): format HHMMSS

• h0 (string): minimum height

• hend (string): maximum height

• MDF (string): name of the MDF used for this case

• noise_fac (string): noise factor

• notes (string): additional notes for the case (stored but not in use by the program)

Parameters:

• data_loc (string) – path to the excel file (make sure the data_loc contains the suffix .xlsx)

• sheet_nr (integer) – number of the desired excel sheet

Returns:

case_list contains the information for all cases

• begin_dt (datetime object): start of the time interval

• end_dt (datetime object): end of the time interval

• plot_range (list): height interval

• MDF_name (string): name of MDF used for this case

• noisefac (string): number of standard deviations above mean noise level

• notes (string): additional notes for the user

pyLARDA.helpers.interp_only_3rd_dim(arr, old, new)

function to interpolate only the velocity (3rd) axis

pyLARDA.helpers.put_in_container(data, data_container, **kwargs)

This routine will generate a new larda container, replacing var in the data_container given with data

Kwargs:

all keys used in larda containers, that is paraminfo (dict): information from params_[campaign].toml for the specific variable rg_unit: range unit mask: ndarray of same size as var, indicating masked values ts: time stamp in unix time var_lims: variable limits colormap: colormap to be used for plotting

Returns:

larda data container

Return type:

container (dict)

pyLARDA.helpers.change_dir(folder_path, **kwargs)

This routine changes to a folder or creates it (including subfolders) if it does not exist already.

Parameters:

folder_path (string) – path of folder to switch into

pyLARDA.helpers.make_dir(folder_path)

This routine changes to a folder or creates it (including subfolders) if it does not exist already.

Parameters:

folder_path (string) – path of folder to switch into

pyLARDA.helpers.print_traceback(txt)

Print the traceback to an error to screen. :param - txt: error msg :type - txt: string

pyLARDA.helpers.smooth(y, box_pts, padding='constant')

Smooth a one dimensional array using a rectangular window of box_pts points

Parameters:

• y (np.array) – array to be smoothed

• box_pts – number of points of the rectangular smoothing window

Returns:

smoothed array

Return type:

y_smooth (np.arrax)

pyLARDA.NcReader

Try to use netcdf4-python

pyLARDA.NcReader.get_time_slicer(ts: array, f: str, time_interval: List[datetime]) → list

get time slicer from the time_interval Following options are available

1. time_interval with [ts_begin, ts_end]

2. only one timestamp is selected and the found right one would be beyond the ts range -> argnearest instead searchsorted

3. only one is timestamp

Parameters:

• ts – unix timestamps as array

• f – filename

• time_interval – time interval to slice for

Returns:

slicer

pyLARDA.NcReader.get_var_attr_from_nc(name, paraminfo, variable)

get the attribute from the variable

Parameters:

• name

• paraminfo

• variable

pyLARDA.NcReader.get_meta_from_nc(ncD, meta_spec, varname)

get some meta data into the data_container

specified within the paraminfo meta.name tags - gattr.name: global attribute with name - vattr.name: variable attribute with name - var.name: additional varable (ideally single value)

Parameters:

• ncD – netCDF file object

• meta_spec – dict with all meta. definition

• varname – name of the variable to load (for vattr)

Returns:

dict with meta data

pyLARDA.NcReader.reader(paraminfo)

build a function for reading in time height data

pyLARDA.NcReader.reader_with_groups(paraminfo)

build a function for reading in time height data

pyLARDA.NcReader.auxreader(paraminfo)

build a function for reading in time height data

pyLARDA.NcReader.timeheightreader_rpgfmcw(paraminfo)

build a function for reading in time height data special function for a special instrument ;)

the issues are:

• range variable in different file

• stacking of single variables

for now works only with 3 chirps and range variable only in level0

pyLARDA.NcReader.specreader_rpgfmcw(paraminfo)

build a function for reading in spectral data special function for a special instrument ;)

the issues are:

• range variable in different file

• stacking of single variables

for now works only with 3 chirps and range variable only in level0

pyLARDA.NcReader.specreader_rpgpy(paraminfo)

build a function for reading in spectral data

pyLARDA.NcReader.scanreader_mira(paraminfo)

reader for the scan files

• load full file regardless of selected time

• covers spec_timeheight and spec_time

pyLARDA.NcReader.interp_only_3rd_dim(arr, old, new, **kwargs)

function to interpolate only the velocity (3rd) axis

pyLARDA.NcReader.specreader_kazr(paraminfo)

build a function for reading in spectral data another special function for another special instrument ;)

the issues are:

• variables time and range are merged and can be accessed by a locator mask

• noise is not saved and has to be computed from the spectra

• spectra are not in reflectivity but in 10*log10(mW)

• need to be converted using specZ = spec * h.z2lin(float(f.cal_constant[:-3])) * self.range[ir]**2

pyLARDA.NcReader.reader_pollyraw(paraminfo)

build a function for reading in the polly raw data into larda

pyLARDA.NcReader.reader_wyoming_sounding(paraminfo)

build a reader to read in Wyoming Upper Air soundings, saved locally as a txt file :param paraminfo: parameter information from toml file

Returns:

reader function

pyLARDA.NcReader.psd_reader(paraminfo)

build a function for reading in in-situ data (particle size distributions)

pyLARDA.NcWrite

pyLARDA.ParameterInfo

pyLARDA.ParameterInfo.deep_update(source, overrides)

Update a nested dictionary. Only additions, no removal modify source in place.

pyLARDA.ParameterInfo.select_matching_template(syskey, templates, fname)

templates either match by exact name or regex in also_match

Example

in a template ``` [CLOUDNET]

also_match = “CLOUDNET_.+” [CLOUDNET.generic] …

` also matches ``CLOUDNET_LIMRAD

pyLARDA.ParameterInfo.check_filter(filter, config)

filter for a specific key in the config file, i.e. instr_name, instr_id, …

Parameters:

• filter – dict with name (instr_name) and matching value (MIRA_NMRA)

• config – dict loaded from the config file

Returns:

True or False

TODO: maybe at one point we will need path level filtering

pyLARDA.ParameterInfo.do_parameter_generic_inheritance(config, cinfo_hand_down={})

inhert the parameter.generic information to the single variables

Parameters:

• config (dict) – config dict as read from toml

• cinfo_hand_down (dict, optional) – campaign information to inherit

Returns:

config

class pyLARDA.ParameterInfo.ParameterInfo(config_path, config_file, cinfo_hand_down={}, root=None)

load the config file right here no need for prior allocation

Parameters:

• config_path – location of the .toml config file

• config_file – name of the .toml config file

iterate_systems(keys=None, filter=None)

provide iterator for the systems structure

Parameters:

keys – just iterate over the given (i.e. valid) systems

Yields:

syskey, config