API documentation¶
pyLARDA main module¶
-
class
pyLARDA.
LARDA
(data_source='local', uri=None)[source]¶ init a new larda instance
Parameters: - data_source (str, optional) – either
'local'
,'remote'
or'filepath'
- uri – link to backend
Returns: larda object
-
connect_local
(camp_name, build_lists=True, filt=None)[source]¶ built the connector list for the specified campaign (only valid systems are considered) the connectors are instances of the Connector.Connector Class
NEW: one connector per system then the params are parts of this system
Parameters: - camp_name (str) – name of campaign as defined in
campaigns.toml
- build_lists (Bool, optional) – Flag to build the filelists or not (with many files this may take some time)
- filt (list, optional) – Filter for name
['system', 'MIRA']
or insturment identifiert['instr_name', 'hatpro_g2_lacros']
- camp_name (str) – name of campaign as defined in
-
read
(system, parameter, time_interval, *further_slices, **kwargs)[source]¶ Parameters: - system (str) – identifier for the system
- parameter (str) – choosen param
- time_interval –
[dt, dt]
time interval, or [dt] one time - *further_slices – range, vel,..
[0, max]
or [3000]
Returns: the dictionary with data
- data_source (str, optional) – either
pyLARDA.Connector¶
-
pyLARDA.Connector.
convert_regex_date_to_dt
(re_date)[source]¶ convert a re_date dict to datetime
Warning
When using 2 digit years (i.e. RPG) a 20 will be added in front
Parameters: re_date (dict) – result of the regex search with keys Returns: datetime
-
pyLARDA.Connector.
convert_to_datestring
(datepattern, f)[source]¶ convert the date in a (file-)string to dt
Parameters: - datepatttern – a python regex definition with named groups
- f – the string
Returns: datetime
-
pyLARDA.Connector.
setup_valid_date_filter
(valid_dates) → Callable[source]¶ validator function for chunks of valid dates
Parameters: valid_dates – list of [begin, end] in ‘YYYYMMDD’ Returns: a single argument (‘YYYYMMDD-HHMMSS’) validator function
-
pyLARDA.Connector.
path_walk
(top, prefilter='.*', topdown=False, followlinks=False)[source]¶ pendant for os.walk
-
pyLARDA.Connector.
guess_end
(dates)[source]¶ estimate the end of a file
Returns: list of pairs [begin, end]
-
class
pyLARDA.Connector.
Connector_remote
(camp_name, system, plain_dict, uri)[source]¶ connect the data (from the a remote source) to larda
Parameters: - camp_name (str) – campaign name
- system (str) – system identifier
- plain_dict (dict) – connector meta info
- uri (str) – address of the remote source
-
collect
(param, time_interval, *further_intervals, **kwargs) → dict[source]¶ collect the data from a parameter for the given intervals
Parameters: - param (str) –
- time_interval – list of begin and end datetime
- *further_intervals – range, velocity, …
- **interp_rg_join – interpolate range during join
Returns: data_container
-
pyLARDA.Connector.
walk_str
(pathinfo)[source]¶ match the names and subdirs with regex using string
only works for unix systems, but is reasonably fast
Parameters: pathinfo – dict Returns: all_files
-
pyLARDA.Connector.
walk_pathlib
(pathinfo)[source]¶ match the names and subdirs with regex using pathlib
should give cross-platform compatability, but comes with a performance penalty (>3x)
Parameters: pathinfo – dict Returns: all_files
-
class
pyLARDA.Connector.
Connector
(system, system_info, valid_dates, description_dir=None)[source]¶ connect the data (from the ncfiles/local sources) to larda
Parameters: - system (str) – system identifier
- system_info (dict) – dict info loaded from toml
- valid_dates (list of lists) – list of begin and end datetime
- description_dir (optional) – dir with the description rst
-
collect
(param, time_interval, *further_intervals, **kwargs) → dict[source]¶ collect the data from a parameter for the given intervals
Parameters: - param (str) –
- time_interval – list of begin and end datetime
- *further_intervals – range, velocity, …
- **interp_rg_join – interpolate range during join
Returns: data_container
-
collect_path
(param, time_interval, *further_intervals, **kwargs) → dict[source]¶ “
Returns: data_container
pyLARDA.helpers¶
-
pyLARDA.helpers.
get_converter_array
(string, **kwargs)[source]¶ colletion of converters that works on arrays combines time, range and varconverters (i see no conceptual separation here)
the maskconverter becomes relevant, if the order is no time, range, whatever (as in mira spec)
chaining example:
`var_conversion = 'z2lin,extrfromaxis2(0)'`
Returns: (varconverter, maskconverter) which both are functions
-
pyLARDA.helpers.
flatten
(xs)[source]¶ flatten inhomogeneous deep lists e.g.
[[1,2,3],4,5,[6,[7,8],9],10]
-
pyLARDA.helpers.
argnearest
(array, value)[source]¶ find the index of the nearest value in a sorted array for example time or range axis
Parameters: - array (np.array) – sorted array with values, list will be converted to 1D array
- value – value to find
Returns: index
-
pyLARDA.helpers.
nearest
(array, pivot)[source]¶ find the nearest value to a given one
Parameters: - array (np.array) – sorted array with values
- pivot – value to find
Returns: value with smallest distance
-
pyLARDA.helpers.
get_extrfromaxis2
(string)[source]¶ get function that extracts given index from axis2
-
pyLARDA.helpers.
fill_with
(array, mask, fill)[source]¶ fill an array where mask is true with fill value
-
pyLARDA.helpers.
guess_str_to_dict
(string)[source]¶ try to convert a text string into a dict intended to be used in the var_def
Returns: dict with flag as key and description string
-
pyLARDA.helpers.
reshape_spectra
(data)[source]¶ This function reshapes time, range and var variables of a data container and returns numpy arrays.
Parameters: data (dict) – data container Returns: list with - ts (numpy.array): time stamp numpy array, dim = (n_time,)
- rg (numpy.array): range stamp numpy array, dim = (n_range,)
- var (numpy.array): values of the spectra numpy array, dim = (n_time, n_range, n_vel)
-
pyLARDA.helpers.
isKthBitSet
(n, k)[source]¶ Function to check if a certain bit of a number is set (required to analyse quality flags)
-
pyLARDA.helpers.
pprint
(data, verbose=False)[source]¶ print a pretty representation of the data container
-
pyLARDA.helpers.
extract_case_from_excel_sheet
(data_loc, sheet_nr=0, **kwargs)[source]¶ This function extracts information from an excel sheet. It can be used for different scenarios. The first row of the excel sheet contains the headline, defined as follows:
A B C D E F G H 1 date start end h0 hend MDF noise_fac notes ORA B C D E F 1 datestart dateend h0 hend noise_fac notes The following rows contain the cases of interest. Make sure that the ALL the data in the excel sheet is formatted as string! The data has to be provided in the following syntax:
- date (string): format YYYYMMDD
- start (string): format HHMMSS
- datestart (string): format YYYYMMDDHHMMSS
- dateend (string): format YYYYMMDDHHMMSS
- end (string): format HHMMSS
- h0 (string): minimum height
- hend (string): maximum height
- MDF (string): name of the MDF used for this case
- noise_fac (string): noise factor
- notes (string): additional notes for the case (stored but not in use by the program)
Parameters: - data_loc (string) – path to the excel file (make sure the data_loc contains the suffix .xlsx)
- sheet_nr (integer) – number of the desired excel sheet
Returns: case_list contains the information for all cases
- begin_dt (datetime object): start of the time interval
- end_dt (datetime object): end of the time interval
- plot_range (list): height interval
- MDF_name (string): name of MDF used for this case
- noisefac (string): number of standard deviations above mean noise level
- notes (string): additional notes for the user
-
pyLARDA.helpers.
interp_only_3rd_dim
(arr, old, new)[source]¶ function to interpolate only the velocity (3rd) axis
-
pyLARDA.helpers.
put_in_container
(data, data_container, **kwargs)[source]¶ This routine will generate a new larda container, replacing var in the data_container given with data
- Kwargs:
- all keys used in larda containers, that is paraminfo (dict): information from params_[campaign].toml for the specific variable rg_unit: range unit mask: ndarray of same size as var, indicating masked values ts: time stamp in unix time var_lims: variable limits colormap: colormap to be used for plotting
Returns: larda data container Return type: container (dict)
-
pyLARDA.helpers.
change_dir
(folder_path, **kwargs)[source]¶ This routine changes to a folder or creates it (including subfolders) if it does not exist already.
Parameters: folder_path (string) – path of folder to switch into
-
pyLARDA.helpers.
make_dir
(folder_path)[source]¶ This routine changes to a folder or creates it (including subfolders) if it does not exist already.
Parameters: folder_path (string) – path of folder to switch into
-
pyLARDA.helpers.
print_traceback
(txt)[source]¶ Print the traceback to an error to screen. :param - txt: error msg :type - txt: string
-
pyLARDA.helpers.
smooth
(y, box_pts, padding='constant')[source]¶ Smooth a one dimensional array using a rectangular window of box_pts points
Parameters: - y (np.array) – array to be smoothed
- box_pts – number of points of the rectangular smoothing window
Returns: smoothed array
Return type: y_smooth (np.arrax)
pyLARDA.NcReader¶
Try to use netcdf4-python
-
pyLARDA.NcReader.
get_time_slicer
(ts: numpy.array, f: str, time_interval: List[datetime.datetime]) → list[source]¶ get time slicer from the time_interval Following options are available
- time_interval with [ts_begin, ts_end]
- only one timestamp is selected and the found right one would be beyond the ts range -> argnearest instead searchsorted
- only one is timestamp
Parameters: - ts – unix timestamps as array
- f – filename
- time_interval – time interval to slice for
Returns: slicer
-
pyLARDA.NcReader.
get_var_attr_from_nc
(name, paraminfo, variable)[source]¶ get the attribute from the variable
Parameters: - name –
- paraminfo –
- variable –
-
pyLARDA.NcReader.
get_meta_from_nc
(ncD, meta_spec, varname)[source]¶ get some meta data into the data_container
specified within the paraminfo meta.name tags - gattr.name: global attribute with name - vattr.name: variable attribute with name - var.name: additional varable (ideally single value)
Parameters: - ncD – netCDF file object
- meta_spec – dict with all meta. definition
- varname – name of the variable to load (for vattr)
Returns: dict with meta data
-
pyLARDA.NcReader.
timeheightreader_rpgfmcw
(paraminfo)[source]¶ build a function for reading in time height data special function for a special instrument ;)
the issues are:
- range variable in different file
- stacking of single variables
for now works only with 3 chirps and range variable only in level0
-
pyLARDA.NcReader.
specreader_rpgfmcw
(paraminfo)[source]¶ build a function for reading in spectral data special function for a special instrument ;)
the issues are:
- range variable in different file
- stacking of single variables
for now works only with 3 chirps and range variable only in level0
-
pyLARDA.NcReader.
scanreader_mira
(paraminfo)[source]¶ reader for the scan files
- load full file regardless of selected time
- covers spec_timeheight and spec_time
-
pyLARDA.NcReader.
interp_only_3rd_dim
(arr, old, new, **kwargs)[source]¶ function to interpolate only the velocity (3rd) axis
-
pyLARDA.NcReader.
specreader_kazr
(paraminfo)[source]¶ build a function for reading in spectral data another special function for another special instrument ;)
the issues are:
- variables time and range are merged and can be accessed by a locator mask
- noise is not saved and has to be computed from the spectra
- spectra are not in reflectivity but in 10*log10(mW)
- need to be converted using specZ = spec * h.z2lin(float(f.cal_constant[:-3])) * self.range[ir]**2
-
pyLARDA.NcReader.
reader_pollyraw
(paraminfo)[source]¶ build a function for reading in the polly raw data into larda
pyLARDA.NcWrite¶
pyLARDA.ParameterInfo¶
-
pyLARDA.ParameterInfo.
deep_update
(source, overrides)[source]¶ Update a nested dictionary. Only additions, no removal modify
source
in place.
-
pyLARDA.ParameterInfo.
select_matching_template
(syskey, templates, fname)[source]¶ templates either match by exact name or regex in also_match
Example
in a template ``` [CLOUDNET]
also_match = “CLOUDNET_.+” [CLOUDNET.generic] …` also matches ``CLOUDNET_LIMRAD
-
pyLARDA.ParameterInfo.
check_filter
(filter, config)[source]¶ filter for a specific key in the config file, i.e. instr_name, instr_id, …
Parameters: - filter – dict with name (instr_name) and matching value (MIRA_NMRA)
- config – dict loaded from the config file
Returns: True or False
TODO: maybe at one point we will need path level filtering
-
pyLARDA.ParameterInfo.
do_parameter_generic_inheritance
(config, cinfo_hand_down={})[source]¶ inhert the parameter.generic information to the single variables
Parameters: - config (dict) – config dict as read from toml
- cinfo_hand_down (dict, optional) – campaign information to inherit
Returns: config