QC checks on individual reports¶

marine_qc.do_valid_value_check(value)[source]

Check if a value is not equal to None or numerically valid.

Parameters:

value (ValueNumberType) – The input value(s) to be tested. Can be a scalar, sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 1 (or array/sequence/Series of 1s) if the input value is None or numerically invalid (NaN)
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays in value_check() does not return np.ndarrays.

marine_qc.do_valid_value_clim_check(climatology, **kwargs)[source]

Check if a climatological value is not equal to None or numerically valid.

Parameters:

climatology (ClimArgType) – The input climatological value(s) to be tested. Can be a scalar, sequence, a one-dimensional NumPy array, a pandas Series, a Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
**kwargs (dict) – Additional keyword arguments passed by the decorator framework (unused).

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 1 (or array/sequence/Series of 1s) if the input value is None or numerically invalid (NaN)
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays in value_check() does not return np.ndarrays.

Notes

If climatology is a Climatology object, pass lon and lat and date, or month and day, as keyword arguments to extract the relevant climatological value.

marine_qc.do_missing_value_check(value)[source]

Check if a value is equal to None or numerically invalid (NaN).

Parameters:

value (ValueNumberType) – The input value(s) to be tested. Can be a scalar, sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 0 (or array/sequence/Series of 1s) if the input value is None or numerically invalid (NaN)
Returns 1 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays in value_check() does not return np.ndarrays.

marine_qc.do_missing_value_clim_check(climatology, **kwargs)[source]

Check if a climatological value is equal to None or numerically invalid (NaN).

Parameters:

climatology (ClimArgType) – The input climatological value(s) to be tested. Can be a scalar, sequence, a one-dimensional NumPy array, a pandas Series, a Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
**kwargs (dict) – Additional keyword arguments passed by the decorator framework (unused).

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 0 (or array/sequence/Series of 1s) if the input value is None or numerically invalid (NaN)
Returns 1 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays in value_check() does not return np.ndarrays.

Notes

If climatology is a Climatology object, pass lon and lat and date, or month and day, as keyword arguments to extract the relevant climatological value.

marine_qc.do_position_check(lat, lon)[source]

Perform the positional QC check on the report.

Simple check to make sure that the latitude and longitude are within specified bounds: - Latitude is between -90 and 90. - Longitude is between 180 and 360.

Parameters:

lat (ValueNumberType) – Latitude(s) of observation in degrees. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
lon (ValueNumberType) – Longitude() of observation in degrees. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if either latitude or longitude is numerically invalid (None/NaN).
Returns 1 (or array/sequence/Series of 1s) if either latitude or longitude is out of the valid range.
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_date_check(date=None, year=None, month=None, day=None, year_init=None, year_end=None)[source]

Perform the date QC check on the report. Checks whether the given date or date components are valid.

Parameters:

date (ValueDatetimeType, optional) – Date(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
year (ValueIntType, optional) – Year(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
month (ValueIntType, optional) – Month(s) of observation (1-12). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
day (ValueIntType, optional) – Day(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
year_init (int, optional) – Initial valid year.
year_end (int, optional) – Last valid year.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if any of year, month, or day is numerically invalid or None,
Returns 1 (or array/sequence/Series of 1s) if the date is not valid,
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_time_check(date=None, hour=None)[source]

Check that the time is valid i.e. in the range 0.0 to 23.99999…

Parameters:

date (ValueDatetimeType, optional) – Date(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
hour (ValueFloatType, optional) – Hour(s) of observation (minutes as decimal). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if hour is numerically invalid or None,
Returns 1 (or array/sequence/Series of 1s) if hour is not a valid hour,
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_day_check(date=None, year=None, month=None, day=None, hour=None, lat=None, lon=None, time_since_sun_above_horizon=None)[source]

Determine if the sun was above the horizon a specified time before the report.

This “day” test is used to classify Marine Air Temperature (MAT) measurements as either Night MAT (NMAT) or Day MAT, accounting for solar heating biases and a potential lag between sun rise and the onset of significant warming. The function calculates the sun’s elevation using the sunangle function, offset by the specified time_since_sun_above_horizon.

Parameters:

date (ValueDatetimeType, optional) – Date(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
year (ValueIntType, optional) – Year(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
month (ValueIntType, optional) – Month(s) of observation (1-12). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
day (ValueIntType, optional) – Day(s) of observation. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
hour (ValueFloatType, optional) – Hour(s) of observation (minutes as decimal). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
lat (ValueNumberType, optional) – Latitude(s) of observation in degrees. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
lon (ValueNumberType, optional) – Longitude() of observation in degree. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
time_since_sun_above_horizon (float) – Maximum time sun can have been above horizon (or below) to still count as night. Original QC test had this set to 1.0 i.e. it was night between one hour after sundown and one hour after sunrise.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if any of do_position_check, do_date_check, or do_time_check returns 2.
Returns 1 (or array/sequence/Series of 1s) if any of do_position_check, do_date_check, or do_time_check returns 1 or if it is night (sun below horizon an hour ago).
Returns 0 if it is day (sun above horizon an hour ago).

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

See also

do_day_check: Determine if the sun was above the horizon an hour ago based on date, time, and position.

Notes

marine_qc.do_hard_limit_check(value, limits)[source]

Check if a value is outside specified limits.

Parameters:

value (ValueNumberType) – The value(s) to be tested against the limits. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
limits (tuple of float) – A tuple of two floats representing the lower and upper limit.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if the upper limit is less than or equal to the lower limit, or if the input is invalid (None or NaN).
Returns 1 (or array/sequence/Series of 1s) if value(s) are outside the specified limits.
Returns 0 (or array/sequence/Series of 0s) if value(s) are within limits.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_climatology_check(value, climatology, maximum_anomaly, standard_deviation='default', standard_deviation_limits=None, lowbar=None)[source]

Climatology check to compare a value with a climatological average within specified anomaly limits.

This check supports optional parameters to customize the comparison.

If standard_deviation is provided, the value is converted into a standardised anomaly. Optionally, if standard deviation is outside the range specified by standard_deviation_limits then standard_deviation is set to whichever of the lower or upper limits is closest. If lowbar is provided, the anomaly must be greater than lowbar to fail regardless of standard_deviation.

Parameters:

value (ValueNumberType) – Value(s) to be compared to climatology. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
climatology (ClimArgType) – The climatological average(s) to which the values(s) will be compared. Can be a scalar, sequence, a one-dimensional NumPy array, a pandas Series, a Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
maximum_anomaly (float) – Largest allowed anomaly. If standard_deviation is provided, this is interpreted as the largest allowed standardised anomaly.
standard_deviation (ClimArgType, default: "default") – The standard deviation(s) used to standardise the anomaly If set to “default”, it is internally treated as 1.0. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series, a Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
standard_deviation_limits (tuple of float, optional) – A tuple of two floats representing the upper and lower limits for standard deviation used in the check.
lowbar (float, optional) – The anomaly must be greater than lowbar to fail regardless of standard deviation.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if standard_deviation_limits[1] is less than or equal to standard_deviation_limits[0], or if maximum_anomaly is less than or equal to 0, or if any of value, climate_normal, or standard_deviation is numerically invalid (None or NaN).
Returns 1 (or array/sequence/Series of 1s) if the difference is outside the specified range.
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

Notes

If either climatology or standard_deviation is a Climatology object, pass lon and lat and date, or month and day, as keyword arguments to extract the relevant climatological value(s).

marine_qc.do_supersaturation_check(dpt, at2)[source]

Perform the super saturation check.

Check if a valid dewpoint temperature is greater than a valid air temperature.

Parameters:

dpt (ValueNumberType) – Dewpoint temperature value(s). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
at2 (ValueNumberType) – Air temperature values(s). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if either dpt or at2 is invalid (None or NaN).
Returns 1 (or array/sequence/Series of 1s) if supersaturation is detected,
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_sst_freeze_check(sst, freezing_point, freeze_check_n_sigma='default', sst_uncertainty='default')[source]

Check input sea-surface temperature(s) to see if it is above freezing.

This is a simple freezing point check made slightly more complex. We want to check if a measurement of SST is above freezing, but there are two problems. First, the freezing point can vary from place to place depending on the salinity of the water. Second, there is uncertainty in SST measurements. If we place a hard cut-off at -1.8C, then we are likely to bias the average of many measurements too high when they are near the freezing point - observational error will push the measurements randomly higher and lower, and this test will trim out the lower tail, thus biasing the result. The inclusion of an SST uncertainty parameter might mitigate that, and we allow that possibility here. Note also that many ships make sea-surface temperature measurements to the nearest whole degree, which in the case of water at or close to freezing would round to -2C and would fail a naive test.

Parameters:

sst (ValueNumberType) – Input sea-surface temperature value(s) to be checked. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
freezing_point (float, optional) – The freezing point of the water.
freeze_check_n_sigma (float, optional, default: "default") – Number of uncertainty standard deviations that sea surface temperature can be below the freezing point before the QC check fails.
sst_uncertainty (float, optional, default: "default") – The uncertainty in the SST value.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if any of sst, freezing_point, sst_uncertainty, or n_sigma is numerically invalid (None or NaN).
Returns 1 (or array/sequence/Series of 1s) if sst is below freezing_point by more than n_sigma times sst_uncertainty.
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

Notes

In previous versions, some parameters had default values:

sst_uncertainty: 0.0

freezing_point: -1.80

n_sigma: 2.0

marine_qc.do_wind_consistency_check(wind_speed, wind_direction)[source]

Test to compare windspeed to winddirection to check if they are consistent.

Zero windspeed should correspond to no particular direction (variable) and wind speeds above a threshold should correspond to a particular direction.

Parameters:

wind_speed (ValueNumberType) – Wind speed value(s). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
wind_direction (ValueNumberType) – Wind direction value(s). Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if either wind_speed or wind_direction is invalid (None or NaN).
Returns 1 (or array/sequence/Series of 1s) if wind_speed and wind_direction are inconsistent,
Returns 0 (or array/sequence/Series of 0s) otherwise.

Raises:

TypeError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_landlocked_check(lat, lon, land_sea_mask, land_flag)[source]

Check input position(s) to determine whether they correspond to a land point.

Parameters:

lat (ValueNumberType) – Latitude(s) of observation in degrees. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
lon (ValueNumberType) – Longitude() of observation in degree. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
land_sea_mask (ClimArgType) – Land-sea classification value(s) to which the latitude and longitude values(s) will be compared. Can be a scalar, sequence, a one-dimensional NumPy array, a pandas Series, Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
land_flag (int) – Integer value in land_sea_mask that denotes a land point.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if either latitude or longitude is numerically invalid (None/NaN).
Returns 1 (or array/sequence/Series of 1s) if the position does not correspond to a land point
Returns 0 (or array/sequence/Series of 0s) otherwise

Raises:

ValueError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_maritime_check(lat, lon, sea_land_mask, sea_flag)[source]

Check input position(s) to determine whether they correspond to a sea point.

Parameters:

lat (ValueNumberType) – Latitude(s) of observation in degrees. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
lon (ValueNumberType) – Longitude() of observation in degree. Can be a scalar, a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series.
sea_land_mask (ClimArgType) – Sea-land classification value(s) to which the latitude and longitude values(s) will be compared. Can be a scalar, sequence, a one-dimensional NumPy array, a pandas Series, a Climatology, a path-like string on disk, a xarray Dataset or a xarray DataArray.
sea_flag (int) – Integer value in sea_land_mask that denotes a sea point.

Return type:

ValueIntType

Returns:

ValueIntType – Same type as input, but with integer values

Returns 2 (or array/sequence/Series of 2s) if either latitude or longitude is numerically invalid (None/NaN).
Returns 1 (or array/sequence/Series of 1s) if latitude and longitude denotes not a sea point
Returns 0 (or array/sequence/Series of 0s) otherwise

Raises:

ValueError – If decorator inspect_arrays does not return np.ndarrays.

marine_qc.do_multiple_individual_check(data, qc_dict=None, preproc_dict=None, return_method='all')[source]

Apply one or more quality-control (QC) functions independently to each row of a DataFrame or Series.

Parameters:

data (pd.Series or pd.DataFrame) – Hashable input data.
qc_dict (Mapping, optional) – Nested QC dictionary. Keys represent arbitrary user-specified names for the checks. The values are dictionaries which contain the keys “func” (name of the QC function), “names” (input data names as keyword arguments, that will be retrieved from data) and, if necessary, “arguments” (the corresponding keyword arguments). For more information see Examples.
preproc_dict (Mapping, optional) – Nested pre-processing dictionary. Keys represent variable names that can be used by qc_dict. The values are dictionaries which contain the keys “func” (name of the pre-processing function), “names” (input data names as keyword arguments, that will be retrieved from data), and “inputs” (list of input-given variables). For more information see Examples.
return_method ({"all", "passed", "failed"}, default: "all") – If “all”, return QC dictionary containing all requested QC check flags. If “passed”: return QC dictionary containing all requested QC check flags until the first check passes. Other QC checks are flagged as unstested (3). If “failed”: return QC dictionary containing all requested QC check flags until the first check fails. Other QC checks are flagged as unstested (3).

Return type:

DataFrame | Series

Returns:

pd.DataFrame or pd.Series – A DataFrame (or Series if the input was a Series) whose columns correspond to the QC names in qc_dict and whose values contain QC flags for each row. Flags depend on the QC functions used.

Raises:

NameError – If a function listed in qc_dict or preproc_dict is not defined. If columns listed in qc_dict or preproc_dict are not available in data.
ValueError – If return_method is not one of [“all”, “passed”, “failed”] If variable names listed in qc_dict or preproc_dict are not valid parameters of the QC function.

Notes

If a variable is pre-processed using preproc_dict, mark the variable name as “__preprocessed__” in qc_dict. For example: “climatology”: “__preprocessed__”.

For more information, see Examples.

Examples

An example qc_dict for a hard limit test:

qc_dict = {
    "hard_limit_check": {
        "func": "do_hard_limit_check",
        "names": "ATEMP",
        "arguments": {"limits": [193.15, 338.15]},
    }
}

An example qc_dict for a climatology test. Variable “climatology” was previously defined:

qc_dict = {
    "climatology_check": {
        "func": "do_climatology_check",
        "names": {
            "value": "observation_value",
            "lat": "latitude",
            "lon": "longitude",
            "date": "date_time",
        },
        "arguments": {
            "climatology": climatology,
            "maximum_anomaly": 10.0,  # K
        },
    },
}

An example preproc_dict for extracting a climatological value:

preproc_dict = {
    "func": "get_climatological_value",
    "names": {
        "lat": "latitude",
        "lon": "longitude",
        "date": "date_time",
    },
    "inputs": climatology,
}

Make use of both dictionaries:

preproc_dict = {
    "func": "get_climatological_value",
    "names": {
        "lat": "latitude",
        "lon": "longitude",
        "date": "date_time",
    },
    "inputs": climatology,
}

qc_dict = {
    "climatology_check": {
        "func": "do_climatology_check",
        "names": {
            "value": "observation_value",
        },
        "arguments": {
            "climatology": "__preprocessed__",
            "maximum_anomaly": 10.0,  # K
        },
    },
}

Finally, run the function:

do_multiple_individual_check(
    data=df,
    qc_dict=qc_dict,
    preproc_dict=preproc_dict,
    return_method="failed",
)