Buoy tracking classes¶
- class marine_qc.buoy_tracking_qc.SpeedChecker(lons, lats, dates, speed_limit, min_win_period, max_win_period)[source]
Class used to carry out
do_speed_check().The check identifies whether a drifter has been picked up by a ship (out of water) based on 1/100th degree precision positions. A flag is set for each input report: flag=1 for reports deemed picked up, else flag=0.
A drifter is deemed picked up if it is moving faster than might be expected for a fast ocean current (a few m/s). Unreasonably fast movement is detected when speed of travel between report-pairs exceeds the chosen ‘speed_limit’ (speed is estimated as distance between reports divided by time separation - this ‘straight line’ speed between the two points is a minimum speed estimate given a less-direct path may have been followed). Positional errors introduced by lon/lat ‘jitter’ and data precision can be of order several km’s. Reports must be separated by a suitably long period of time (the ‘min_win_period’) to minimise the effect of these errors when calculating speed e.g. for reports separated by 24 hours errors of several cm/s would result which are two orders of magnitude less than a fast ocean current which seems reasonable. Conversely, the period of time chosen should not be too long so as to resolve short-lived burst of speed on manoeuvring ships. Larger positional errors may also trigger the check. Because temporal sampling can be erratic the time period over which this assessment is made is specified as a range (bound by ‘min_win_period’ and ‘max_win_period’) - assessment uses the longest time separation available within this range.
IMPORTANT - for optimal performance, drifter records with observations failing this check should be subsequently manually reviewed. Ships move around in all sorts of complicated ways that can readily confuse such a simple check (e.g. pausing at sea, crisscrossing its own path) and once some erroneous movement is detected it is likely a human operator can then better pick out the actual bad data. False fails caused by positional errors (particularly in fast ocean currents) will also need reinstating.
- Parameters:
lons (
SequenceNumberType) – 1-dimensional longitude array in degrees.lats (
SequenceNumberType) – 1-dimensional latitude array in degrees.dates (
SequenceDatetimeType) – 1-dimensional date array.speed_limit (
float) – Maximum allowable speed for an in situ drifting buoy (metres per second).min_win_period (
float) – Minimum period of time in days over which position is assessed for speed estimates (see description).max_win_period (
float) – Maximum period of time in days over which position is assessed for speed estimates (this should be greater than min_win_period and allow for some erratic temporal sampling e.g. min_win_period + 0.2 to allow for gaps of up to 0.2 - days in sampling).
- get_qc_outcomes()[source]
Retrieve the QC outcomes for all observations.
- Return type:
- Returns:
np.ndarray– Array of QC flags for each observation (0 = passed, 1 = failed, untested otherwise).
- valid_arrays()[source]
Validate the input observation arrays (longitude, latitude, and time differences).
Checks for: - NaN values in longitude, latitude, or time arrays. - Monotonicity of the time array (self.hrs).
Warnings are raised for any issues detected.
- valid_parameters()[source]
Validate the QC parameters to ensure they are sensible.
Checks that: - speed_limit is non-negative. - min_win_period is non-negative. - max_win_period is greater than or equal to min_win_period.
Warnings are raised for any invalid parameters.
- class marine_qc.buoy_tracking_qc.NewSpeedChecker(lons, lats, dates, speed_limit, min_win_period, ship_speed_limit, delta_d, delta_t, n_neighbours)[source]
Class used to carry out
do_new_speed_check().Check to see whether a drifter has been picked up by a ship (out of water) based on 1/100th degree precision positions. A flag is set for each input report: flag=1 for reports deemed picked up, else flag=0.
A drifter is deemed picked up if it is moving faster than might be expected for a fast ocean current (a few m/s). Unreasonably fast movement is detected when speed of travel between report-pairs exceeds the chosen ‘speed_limit’ (speed is estimated as distance between reports divided by time separation - this ‘straight line’ speed between the two points is a minimum speed estimate given a less-direct path may have been followed). Positional errors introduced by lon/lat ‘jitter’ and data precision can be of order several km’s. Reports must be separated by a suitably long period of time (the ‘min_win_period’) to minimise the effect of these errors when calculating speed e.g. for reports separated by 9 hours errors of order 10 cm/s would result which are a few percent of fast ocean current speed. Conversely, the period of time chosen should not be too long so as to resolve short-lived burst of speed on manouvering ships. Larger positional errors may also trigger the check.
For each report, speed is assessed over the shortest available period that exceeds ‘min_win_period’.
Prior to assessment the drifter record is screened for positional errors using the iQuam track check method (from
ex.Voyage). When running the iQuam check the record is treated as a ship (not a drifter) so as to avoid accidentally filtering out observations made aboard a ship (which is what we are trying to detect). This iQuam track check does not overwrite any existing iQuam track check flags.IMPORTANT - for optimal performance, drifter records with observations failing this check should be subsequently manually reviewed. Ships move around in all sorts of complicated ways that can readily confuse such a simple check (e.g. pausing at sea, crisscrossing its own path) and once some erroneous movement is detected it is likely a human operator can then better pick out the actual bad data. False fails caused by positional errors (particularly in fast ocean currents) will also need reinstating.
The class has the following class attributes which can be modified using the set_parameters method.
iquam_parameters: Parameter dictionary for Voyage.iquam_track_check() function.
speed_limit: maximum allowable speed for an in situ drifting buoy (metres per second)
min_win_period: minimum period of time in days over which position is assessed for speed estimates (see description)
- Parameters:
lons (
SequenceNumberType) – 1-dimensional longitude array in degrees.lats (
SequenceNumberType) – 1-dimensional latitude array in degrees.dates (
SequenceDatetimeType) – 1-dimensional date array.speed_limit (
float) – Maximum allowable speed for an in situ drifting buoy (metres per second).min_win_period (
float) – Minimum period of time in days over which position is assessed for speed estimates (see description).ship_speed_limit (
float) – Ship speed limit for the IQUAM track check.delta_d (
float) – The smallest increment in distance that can be resolved. For 0.01 degrees of lat-lon this is 1.11 km. Used in the IQUAM track check.delta_t (
float) – The smallest increment in time that can be resolved. For hourly data expressed as a float this is 0.01 hours. Used in the IQUAM track check.n_neighbours (
int) – Number of neighbours considered in the IQUAM track check.
- get_qc_outcomes()[source]
Retrieve the QC outcomes for all observations.
- Return type:
- Returns:
np.ndarray– Array of QC flags for each observation (0 = valid, 1 = flagged, untested otherwise).
- perform_iquam_track_check()[source]
Perform iQuam track check as if reports are from a ship.
A deep copy of reps is made so metadata can be safely modified ahead of iQuam check an array of qc flags (iquam_track_ship) is the result.
- Return type:
- valid_arrays()[source]
Validate the input arrays (longitude, latitude, and time differences).
Checks for: - NaN values in longitude, latitude, or time arrays. - Monotonicity of the time array (self.hrs).
Warnings are raised for any issues detected.
- class marine_qc.buoy_tracking_qc.AgroundChecker(lons, lats, dates, smooth_win, min_win_period, max_win_period)[source]
Class used to carry out
do_aground_check().Check to see whether a drifter has run aground based on 1/100th degree precision positions. A flag is set for each input report: flag=1 for reports deemed aground, else flag=0.
Positional errors introduced by lon/lat ‘jitter’ and data precision can be of order several km’s. Longitude and latitude timeseries are smoothed prior to assessment to reduce position ‘jitter’. Some post-smoothing position ‘jitter’ may remain and its expected magnitude is set within the function by the ‘tolerance’ parameter. A drifter is deemed aground when, after a period of time, the distance between reports is less than the ‘tolerance’. The minimum period of time over which this assessment is made is set by ‘min_win_period’. This period must be long enough such that slow moving drifters are not falsely flagged as aground given errors in position (e.g. a buoy drifting at around 1 cm/s will travel around 1 km/day; given ‘tolerance’ and precision errors of a few km’s the ‘min_win_period’ needs to be several days to ensure distance-travelled exceeds the error so that motion is reliably detected and the buoy is not falsely flagged as aground). However, min_win_period should not be longer than necessary as buoys that run aground for less than min_win_period will not be detected.
Because temporal sampling can be erratic the time period over which an assessment is made is specified as a range (bound by ‘min_win_period’ and ‘max_win_period’) - assessment uses the longest time separation available within this range. If a drifter is deemed aground and subsequently starts moving (e.g. if a drifter has moved very slowly for a prolonged period) incorrectly flagged reports will be reinstated.
smooth_win: length of window (odd number) in datapoints used for smoothing lon/lat
min_win_period: minimum period of time in days over which position is assessed for no movement (see description)
max_win_period: maximum period of time in days over which position is assessed for no movement (this should be greater than min_win_period and allow for erratic temporal sampling e.g. min_win_period+2 to allow for gaps of up to 2-days in sampling).
- Parameters:
lons (
SequenceNumberType) – 1-dimensional longitude array in degrees.lats (
SequenceNumberType) – 1-dimensional latitude array in degrees.dates (
SequenceDatetimeType) – 1-dimensional date array.smooth_win (
int) – Length of window (odd number) in datapoints used for smoothing lon/lat.min_win_period (
int) – Minimum period of time in days over which position is assessed for no movement (see description).max_win_period (
intorNone) – Maximum period of time in days over which position is assessed for no movement (this should be greater than min_win_period and allow for erratic temporal sampling e.g. min_win_period+2 to allow for gaps of up to 2-days in sampling).
- get_qc_outcomes()[source]
Return the QC outcomes.
- tolerance = 1.5725359013624185
- valid_arrays()[source]
Check the input arrays are valid. Raises a warning and returns False if not valid.
- class marine_qc.buoy_tracking_qc.SSTTailChecker(lat, lon, sst, ostia, ice, bgvar, dates, long_win_len, long_err_std_n, short_win_len, short_err_std_n, short_win_n_bad, drif_inter, drif_intra, background_err_lim)[source]
Class used to carry out
do_sst_start_tail_check()anddo_sst_end_tail_check().Check to see whether there is erroneous sea surface temperature data at the beginning or end of a drifter record (referred to as ‘tails’). Flags are set for each input report: flag=1 for reports with erroneous data, else flag=0, ‘drf_tail1’ is used for bad data at the beginning of a record, ‘drf_tail2’ is used for bad data at the end of a record.
The tail check makes an assessment of the quality of data at the start and end of a drifting buoy record by comparing to a background reference field. Data found to be unacceptably biased or noisy relative to the background are flagged by the check. When making the comparison an allowance is made for background error variance and also normal drifter error (both bias and random measurement error). The correlation of the background error is treated as unknown and takes on a value which maximises background error dependent on the assessment being made. A background error variance limit is also specified, beyond which the background is deemed unreliable. Observations made during the day, in icy regions or where the background value is missing are excluded from the comparison.
The check proceeds in two steps; a ‘long tail-check’ followed by a ‘short tail-check’. The idea is that the short tail-check has finer resolution but lower sensitivity than the long tail-check and may pick off noisy data not picked up by the long tail check. Only observations that pass the long tail-check are passed to the short tail-check. Both of these tail checks proceed by moving a window over the data and assessing the data in each window. Once good data are found the check stops and any bad data preceding this are flagged. If unreliable background data are encountered the check stops. The checks are run forwards and backwards over the record so as to assess data at the start and end of the record. If the whole record fails no observations are flagged as there are then no ‘tails’ in the data (this is left for other checks). The long tail check looks for groups of observations that are too biased or noisy as a whole. The short tail check looks for individual observations exceeding a noise limit within the window.
- Parameters:
lat (
SequenceNumberType) – 1-dimensional latitude array in degrees.lon (
SequenceNumberType) – 1-dimensional longitude array in degrees.sst (
SequenceNumberType) – 1-dimensional array of sea surface temperatures in K.ostia (
SequenceNumberType) – 1-dimensional array of background field sea surface temperatures in K.ice (
SequenceNumberType) – 1-dimensional array of ice concentrations in the range 0.0 to 1.0.bgvar (
SequenceNumberType) – 1-dimensional array of background sea surface temperature fields variances in K^2.dates (
SequenceDatetimeType) – 1-dimensional date array.long_win_len (
int) – Length of window (in data-points) over which to make long tail-check (must be an odd number).long_err_std_n (
float) – Number of standard deviations of combined background and drifter bias error, beyond which data fail bias check.short_win_len (
int) – Length of window (in data-points) over which to make the short tail-check.short_err_std_n (
float) – Number of standard deviations of combined background and drifter error, beyond which data are deemed suspicious.short_win_n_bad (
int) – Minimum number of suspicious data points required for failure of short check window.drif_inter (
float) – Spread of biases expected in drifter data (standard deviation, degC or K).drif_intra (
float) – Maximum random measurement uncertainty reasonably expected in drifter data (standard deviation, degC or K).background_err_lim (
float) – Background error variance beyond which the SST background is deemed unreliable (degC squared).
- bgerr: ndarray
- do_sst_tail_check(start_tail)[source]
Perform the actual SST tail check.
- end_tail_ind: int
- get_qc_outcomes()[source]
Return the QC outcomes.
- reps_ind: ndarray
- sst_anom: ndarray
- start_tail_ind: int
- class marine_qc.buoy_tracking_qc.SSTBiasedNoisyChecker(lat, lon, dates, sst, ostia, bgvar, ice, n_eval, bias_lim, drif_intra, drif_inter, err_std_n, n_bad, background_err_lim)[source]
Class used to perform the
do_sst_biased_check(),do_sst_noisy_check(), anddo_sst_biased_noisy_short_check().Check to see whether a drifter sea surface temperature record is unacceptably biased or noisy as a whole.
The check makes an assessment of the quality of data in a drifting buoy record by comparing to a background reference field. If the record is found to be unacceptably biased or noisy relative to the background all observations are flagged by the check. For longer records the flags ‘drf_bias’ and ‘drf_noise’ are set for each input report: flag=1 for records with erroneous data, else flag=0. For shorter records ‘drf_short’ is set for each input report: flag=1 for reports with erroneous data, else flag=0.
When making the comparison an allowance is made for background error variance and also normal drifter error (both bias and random measurement error). A background error variance limit is also specified, beyond which the background is deemed unreliable and is excluded from comparison. Observations made during the day, in icy regions or where the background value is missing are also excluded from the comparison.
The check has two separate streams; a ‘long-record check’ and a ‘short-record check’. Records with at least n_eval observations are passed to the long-record check, else they are passed to the short-record check. The long-record check looks for records that are too biased or noisy as a whole. The short record check looks for individual observations exceeding a noise limit within a record. The purpose of n_eval is to ensure records with too few observations for their bias and noise to be reliably estimated are handled separately by the short-record check.
The correlation of the background error is treated as unknown and handled differently for each assessment. For the long-record noise-check and the short-record check the background error is treated as uncorrelated, which maximises the possible impact of background error on these assessments. For the long-record bias-check a limit (bias_lim) is specified beyond which the record is considered biased. The default value for this limit was chosen based on histograms of drifter-background bias. An alternative approach would be to treat the background error as entirely correlated across a long-record, which maximises its possible impact on the bias assessment. In this case the histogram approach was used as the limit could be tuned to give better results.
- Parameters:
lat (
SequenceNumberType) – 1-dimensional latitude array in degrees.lon (
SequenceNumberType) – 1-dimensional longitude array in degrees.dates (
SequenceDatetimeType) – 1-dimensional date array.sst (
SequenceNumberType) – 1-dimensional sea surface temperature array in K.ostia (
SequenceNumberType) – 1-dimensional background field sea surface temperature array in K.bgvar (
SequenceNumberType) – 1-dimensional background variance array in K^2.ice (
SequenceNumberType) – 1-dimensional ice concentration array in range 0,1.n_eval (
int) – The minimum number of drifter observations required to be assessed by the long-record check.bias_lim (
float) – Maximum allowable drifter-background bias, beyond which a record is considered biased (degC or K).drif_intra (
float) – Maximum random measurement uncertainty reasonably expected in drifter data (standard deviation, degC or K).drif_inter (
float) – Spread of biases expected in drifter data (standard deviation, degC or K).err_std_n (
float) – Number of standard deviations of combined background and drifter error, beyond which short-record data are deemed suspicious.n_bad (
int) – Minimum number of suspicious data points required for failure of short-record check.background_err_lim (
float) – Background error variance beyond which the SST background is deemed unreliable (degC squared or K squared).
- bgerr: ndarray
- bgvar_is_masked: bool
- get_qc_outcomes_bias()[source]
Return the QC outcomes for the bias check.
- get_qc_outcomes_noise()[source]
Return the QC outcomes for the noisy check.
- get_qc_outcomes_short()[source]
Return the QC outcomes for the short check.
- set_all_qc_outcomes_to(input_state)[source]
Set all the QC outcomes to the specified input_state.
- sst_anom: ndarray
Buoy tracking functions¶
- marine_qc.buoy_tracking_qc.track_day_test(year, month, day, hour, lat, lon, elevdlim=-2.5)[source]
Given date, time, lat and lon calculate if the sun elevation is > elevdlim.
This is the “day” test used by tracking QC to decide whether an SST measurement is night or day. This is important because daytime diurnal heating can affect comparison with an SST background. It uses the function sunangle to calculate the elevation of the sun. A default solar_zenith angle of 92.5 degrees (elevation of -2.5 degrees) delimits night from day.
- Parameters:
- Return type:
- Returns:
bool– True if daytime, else False.- Raises:
ValueError – If either year, month, day, hour, lat or lon is numerically invalid or None of if either month, day, hour or lat is not in valid range.
- marine_qc.buoy_tracking_qc.is_monotonic(inarr)[source]
Test if elements in an array are increasing monotonically.
I.e. each element is greater than or equal to the preceding element.