Function to detect potentially duplicated observations¶

marine_qc.duplicate_check(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, ignore_columns=None, ignore_entries=None, ignore_nan_both=True, ignore_nan_either=None, offsets=None, compare_level_libraries=None, reindex_by_null=True, null_label='null', **kwargs)

Detect potentially duplicated observations using Python SPLINK Toolkit.

This function builds a pandas DataFrame from the provided observation metadata and compares records using configurable record linkage rules.

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The result is returned as a DupDetect object containing the processed input data, detected duplicate groups, and comparison configuration information.

Parameters:

station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.
data (pandas.DataFrame, optional) – A pandas DataFrame containing the relevant input data. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.
ignore_columns (str or list, optional) – Column names to exclude entirely from duplicate detection. For rows containing ignored values, the corresponding comparison is treated as a match for that column.
ignore_entries (dict, optional) – Ignore specific values for selected columns during comparison.

This is useful when placeholder or invalid values should not prevent duplicate matches.

Keys correspond to column names and values correspond to entries to ignore.

Ignore missing station IDs:
```
ignore_entries = {
    "station_id": "UNKNOWN",
}
```
Ignore multiple values:
```
ignore_entries = {
    "station_id": ["UNKNOWN", "MISSING"],
}
```
ignore_nan_both (str, list of str or bool, default: True) – For selected columns, consider two observations as duplicates whenever both compared value are NaN. If True, all columns are affected.
ignore_nan_either (str, list of str or bool, optional) – For selected columns, consider two observations as duplicates whenever either compared value is NaN. If True, all columns are affected.
offsets (dict, optional) – Override comparison offsets for selected columns. This modifies the tolerance used during comparison.

Example:
```
offsets = {
    "lat": 0.05,
    "lon": 0.05,
}
```
compare_level_libraries (dict, optional) – Override comparison levels for selected columns. This modifies the comparison level used during comparison.

Example:
```
compare_level_libraries = {
    "lat: "ExactMatchLevel",
}
```
reindex_by_null (bool, optional) – If True, rows are reordered according to the number of missing values. Rows may be reordered based on the distribution of missing values. This can improve duplicate matching performance and consistency when null values are present.
null_label (str, optional) – Placeholder value used internally when reindex_by_null=True.

**kwargs (dict) – Additional columns to include in duplicate detection.

Extra keyword arguments are added directly to the internal DataFrame.

Example:

duplicate_check(
    station_id,
    lat,
    lon,
    date,
    vsi,
    dsi,
    platform_type=platform_type,
    source=source,
)

Return type:

DupDetect

Returns:

DupDetect – Duplicate detection result object.

The returned object contains:

The processed input data.
Detected duplicate groups.
Comparison configuration information.

Warning

If ignore_nan_either is set, this can lead to misleading duplicate chains.

In this example, we focus on only two input variables:

df = pd.DataFrame(
    {
        "station_id": ["A", "A", "B", "A", None],
        "lon": [29.7, -29.7, 29.7, np.nan, 29.7],
    }
)
print(df)

  station_id   lon
        A  29.7
        A -29.7
        B  29.7
        A   NaN
     None  29.7

This produces the following duplicate pairs:

detected = duplicate_check(data=df, ignore_nan_either=True)

(0,3): ([“A”, 29.7 ], [“A” , np.nan])
(0,4): ([“A”, 29.7 ], [None, 29.7])
(1,3): ([“A”, -29.7 ], [“A” , np.nan])
(2,4): ([“B”, 29.7 ], [None, 29.7])
(3,4): ([“A”, np.nan], [None, 29.7])

All of these duplicates pairs are reasonable if two observations are considered as duplicates whenever either compared value is NaN.

However, this also produces the following misleading duplicate chains, even though the connected observations are clearly not duplicates:

0 -> 3 -> 1
0 -> 4 -> 2

Since (3,4) is also considered a duplicate pair, all entries are connected, resulting in:

print(detected.groups)
>>> [[0, 3, 4, 1, 2]]

Examples

Basic usage:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Using additional observations:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     temperature=temperature,
...     salinity=salinity,
... )

Ignoring placeholder station IDs:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     ignore_entries={"station_id": "UNKNOWN"},
... )

Increasing spatial tolerances:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     offsets={
...         "lat": 1.0,
...         "lon": 1.0,
...     },
... )

marine_qc.get_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Get potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function returns the indices of the detected duplicate observations according to the selected keep strategy.

Parameters:

station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.
detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.
keep (str or int, default: first) – Determines which duplicate entry should be retained.
- "first" keeps the first occurrence.
- "last" keeps the last occurrence.
- Integer values keep the specified positional match.
**kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

Sequence[Any] | ndarray[tuple[Any, ...], dtype[Any]] | Series | ndarray

Returns:

SequenceValueType –

Same type as input, but with indexes’ type values

Returns the indexes of the corresponding duplicate(s).

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

If detected is set, station_id, lat, lon, date, vsi, dsi and data are ignored. If detected is set, the function always returns pandas.Series. If data is set, station_id, lat, lon, date, vsi and dsi are ignored.

Examples

Get duplicates directly from raw observations:

>>> duplicates = get_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … duplicates = get_duplicates(detected=detected)

marine_qc.flag_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Flag potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function returns duplicate flags for the detected observations according to the selected keep strategy.

Parameters:

station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.
detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.
keep (str or int, default: first) – Determines which duplicate entry should be retained.
- "first" keeps the first occurrence.
- "last" keeps the last occurrence.
- Integer values keep the specified positional match.
**kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

SequenceIntType

Returns:

SequenceIntType –

Same type as input, but with integer values

Returns 0 (or array/sequence/Series of 1s) for unique observation(s)

Returns 1 (or array/sequence/Series of 1s) for best duplicate(s)

Returns 3 (or array/sequence/Series of 1s) for worst duplicate(s)

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

Examples

Flag duplicates directly from raw observations:

>>> flags = flag_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … flags = flag_duplicates(detected=detected)

marine_qc.remove_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Remove potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function removes duplicate observations according to the selected keep strategy and returns the filtered input data.

Parameters:

station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.
data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.
detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.
keep (str or int, default: first) – Determines which duplicate entry should be retained.
- "first" keeps the first occurrence.
- "last" keeps the last occurrence.
- Integer values keep the specified positional match.
**kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

Returns:

tuple of result arrays or pd.DataFrame. – Same type as input A tuple of all input data without the removed duplicated rows or a pandas.DataFrame if data is provided.

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

Examples

Remove duplicates directly from raw observations:

>>> results = remove_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … results = remove_duplicates(detected=detected)