marine_qc.duplicate_checker package

Marine Duplicate Checker package.

marine_qc.duplicate_checker.duplicate_check(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, ignore_columns=None, ignore_entries=None, ignore_nan_both=True, ignore_nan_either=None, offsets=None, compare_level_libraries=None, reindex_by_null=True, null_label='null', **kwargs)

Detect potentially duplicated observations using Python SPLINK Toolkit.

This function builds a pandas DataFrame from the provided observation metadata and compares records using configurable record linkage rules.

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The result is returned as a DupDetect object containing the processed input data, detected duplicate groups, and comparison configuration information.

Parameters:

  • station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if data is provided.

  • data (pandas.DataFrame, optional) – A pandas DataFrame containing the relevant input data. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.

  • ignore_columns (str or list, optional) – Column names to exclude entirely from duplicate detection. For rows containing ignored values, the corresponding comparison is treated as a match for that column.

  • ignore_entries (dict, optional) – Ignore specific values for selected columns during comparison.

    This is useful when placeholder or invalid values should not prevent duplicate matches.

    Keys correspond to column names and values correspond to entries to ignore.

    Ignore missing station IDs:

    ignore_entries = {
    "station_id": "UNKNOWN",
}

    Ignore multiple values:

    ignore_entries = {
    "station_id": ["UNKNOWN", "MISSING"],
}

  • ignore_nan_both (str, list of str or bool, default: True) – For selected columns, consider two observations as duplicates whenever both compared value are NaN. If True, all columns are affected.

  • ignore_nan_either (str, list of str or bool, optional) – For selected columns, consider two observations as duplicates whenever either compared value is NaN. If True, all columns are affected.

  • offsets (dict, optional) – Override comparison offsets for selected columns. This modifies the tolerance used during comparison.

    Example:

    offsets = {
    "lat": 0.05,
    "lon": 0.05,
}

  • compare_level_libraries (dict, optional) – Override comparison levels for selected columns. This modifies the comparison level used during comparison.

    Example:

    compare_level_libraries = {
    "lat: "ExactMatchLevel",
}

  • reindex_by_null (bool, optional) – If True, rows are reordered according to the number of missing values. Rows may be reordered based on the distribution of missing values. This can improve duplicate matching performance and consistency when null values are present.

  • null_label (str, optional) – Placeholder value used internally when reindex_by_null=True.

  • **kwargs (dict) – Additional columns to include in duplicate detection.

    Extra keyword arguments are added directly to the internal DataFrame.

    Example:

    duplicate_check(
    station_id,
    lat,
    lon,
    date,
    vsi,
    dsi,
    platform_type=platform_type,
    source=source,
)
Return type:

DupDetect

Returns:

DupDetect – Duplicate detection result object.

The returned object contains:

  • The processed input data.

  • Detected duplicate groups.

  • Comparison configuration information.

Warning

If ignore_nan_either is set, this can lead to misleading duplicate chains.

In this example, we focus on only two input variables:

df = pd.DataFrame(
    {
        "station_id": ["A", "A", "B", "A", None],
        "lon": [29.7, -29.7, 29.7, np.nan, 29.7],
    }
)
print(df)

  station_id   lon
0          A  29.7
1          A -29.7
2          B  29.7
3          A   NaN
4       None  29.7

This produces the following duplicate pairs:

detected = duplicate_check(data=df, ignore_nan_either=True)

  • (0,3): ([“A”, 29.7 ], [“A” , np.nan])

  • (0,4): ([“A”, 29.7 ], [None, 29.7])

  • (1,3): ([“A”, -29.7 ], [“A” , np.nan])

  • (2,4): ([“B”, 29.7 ], [None, 29.7])

  • (3,4): ([“A”, np.nan], [None, 29.7])

All of these duplicates pairs are reasonable if two observations are considered as duplicates whenever either compared value is NaN.

However, this also produces the following misleading duplicate chains, even though the connected observations are clearly not duplicates:

  • 0 -> 3 -> 1

  • 0 -> 4 -> 2

Since (3,4) is also considered a duplicate pair, all entries are connected, resulting in:

print(detected.groups)
>>> [[0, 3, 4, 1, 2]]

Examples

Basic usage:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Using additional observations:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     temperature=temperature,
...     salinity=salinity,
... )

Ignoring placeholder station IDs:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     ignore_entries={"station_id": "UNKNOWN"},
... )

Increasing spatial tolerances:

>>> dup = duplicate_check(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
...     offsets={
...         "lat": 1.0,
...         "lon": 1.0,
...     },
... )
marine_qc.duplicate_checker.flag_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Flag potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function returns duplicate flags for the detected observations according to the selected keep strategy.

Parameters:

  • station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.

  • detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.

  • keep (str or int, default: first) – Determines which duplicate entry should be retained.

    • "first" keeps the first occurrence.

    • "last" keeps the last occurrence.

    • Integer values keep the specified positional match.

  • **kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

SequenceIntType

Returns:

SequenceIntType –

Same type as input, but with integer values

  • Returns 0 (or array/sequence/Series of 1s) for unique observation(s)

  • Returns 1 (or array/sequence/Series of 1s) for best duplicate(s)

  • Returns 3 (or array/sequence/Series of 1s) for worst duplicate(s)

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

If detected is set, station_id, lat, lon, date, vsi, dsi and data are ignored. If detected is set, the function always returns pandas.Series. If data is set, station_id, lat, lon, date, vsi and dsi are ignored.

Examples

Flag duplicates directly from raw observations:

>>> flags = flag_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … flags = flag_duplicates(detected=detected)

marine_qc.duplicate_checker.get_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Get potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function returns the indices of the detected duplicate observations according to the selected keep strategy.

Parameters:

  • station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.

  • detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.

  • keep (str or int, default: first) – Determines which duplicate entry should be retained.

    • "first" keeps the first occurrence.

    • "last" keeps the last occurrence.

    • Integer values keep the specified positional match.

  • **kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

Sequence[Any] | ndarray[tuple[Any, ...], dtype[Any]] | Series | ndarray

Returns:

SequenceValueType –

Same type as input, but with indexes’ type values

Returns the indexes of the corresponding duplicate(s).

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

If detected is set, station_id, lat, lon, date, vsi, dsi and data are ignored. If detected is set, the function always returns pandas.Series. If data is set, station_id, lat, lon, date, vsi and dsi are ignored.

Examples

Get duplicates directly from raw observations:

>>> duplicates = get_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … duplicates = get_duplicates(detected=detected)

marine_qc.duplicate_checker.remove_duplicates(station_id=None, lat=None, lon=None, date=None, vsi=None, dsi=None, data=None, detected=None, keep='first', **kwargs)

Remove potentially duplicated observations using Python SPLINK Toolkit.

This function identifies duplicate observations either from a precomputed DupDetect instance or by internally calling duplicate_check().

Candidate record pairs are generated using the SPLINK framework, after which only pairs satisfying all configured comparison conditions are retained as duplicates.

The function removes duplicate observations according to the selected keep strategy and returns the filtered input data.

Parameters:

  • station_id (SequenceStrType, optional) – One-dimensional array of station IDs. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • date (SequenceDatetimeType, optional) – One-dimensional array of datetime values. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees. Can be a sequence (e.g., list or tuple), a one-dimensional NumPy array, or a pandas Series. Ignored if detected or data is provided.

  • data (pandas.DataFrame, optional) – A pandas.DataFrame containing the relevant input data. Ignored if detected is provided. If provided, all input data arguments (station_id, lat, lon, date, vsi and dsi) are ignored.

  • detected (DupDetect, optional) – A DupDetect instance that already contains detected duplicates to flag. If provided, duplicate detection is not rerun and all input data arguments (station_id, lat, lon, date, vsi, dsi, and data) are ignored.

  • keep (str or int, default: first) – Determines which duplicate entry should be retained.

    • "first" keeps the first occurrence.

    • "last" keeps the last occurrence.

    • Integer values keep the specified positional match.

  • **kwargs (Any) – Additional keyword arguments passed to duplicate_check() when detected is not provided. Additionally, that could be extra input data as well.

Return type:

tuple[Sequence[str | None] | ndarray[tuple[Any, ...], dtype[str_]] | Series | ndarray, SequenceNumberType, SequenceNumberType, SequenceDatetimeType, SequenceNumberType, SequenceNumberType, Unpack[tuple[Sequence[Any] | ndarray[tuple[Any, ...], dtype[Any]] | Series | ndarray, ...]]] | DataFrame

Returns:

tuple of result arrays or pd.DataFrame. – Same type as input A tuple of all input data without the removed duplicated rows or a pandas.DataFrame if data is provided.

Raises:

ValueError – If none of detected, data, station_id, lat, lon, date, vsi and dsi is set.

Notes

If detected is set, station_id, lat, lon, date, vsi, dsi and data are ignored. If detected is set, the function always returns pandas.Series. If data is set, station_id, lat, lon, date, vsi and dsi are ignored.

Examples

Remove duplicates directly from raw observations:

>>> results = remove_duplicates(
...     station_id=station_id,
...     lat=lat,
...     lon=lon,
...     date=date,
...     vsi=vsi,
...     dsi=dsi,
... )

Use a precomputed duplicate detection result: >>> detected = duplicate_check( … station_id=station_id, … lat=lat, … lon=lon, … date=date, … vsi=vsi, … dsi=dsi, … ) … results = remove_duplicates(detected=detected)

Submodules

marine_qc.duplicate_checker.duplicates module

Common Data Model (CDM) pandas duplicate check.

class marine_qc.duplicate_checker.duplicates.DupDetect(groups, settings, data)[source]

Bases: object

Class to detect, flag, and remove duplicate entries in a DataFrame using a comparison matrix from splink.

Parameters:

  • groups (list of list of Any) – Groups of index pairs of potentially duplicated observations.

  • settings (dict) – Settings dict used for duplicate detection.

  • data (pandas.DataFrame) – Original dataset.

flag_duplicates(keep='first')[source]

Get result dataset with flagged duplicates.

Parameters:

keep (str or int, default: first) – Determines which duplicate entry should be retained.

  • "first" keeps the first occurrence.

  • "last" keeps the last occurrence.

  • Integer values keep the specified positional match.

Return type:

Series

Returns:

pandas.Series – Series containing duplicate flags for the detected observations.

References

get_duplicates(keep='first', overwrite=True)[source]

Identify duplicate matches based on the comparison matrix.

Parameters:

  • keep (str or int, default: first) – Determines which duplicate entry should be retained.

    • "first" keeps the first occurrence.

    • "last" keeps the last occurrence.

    • Integer values keep the specified positional match.

  • overwrite (bool, default: True) – Whether to recompute matches if already calculated.

Return type:

Series

Returns:

pandas.Series – A pandas Series containing the indexes of the corresponding duplicate(s).

remove_duplicates(keep='first')[source]

Remove duplicate entries from the dataset.

Parameters:

keep (str or int) – Determines which duplicate entry should be retained.

  • "first" keeps the first occurrence.

  • "last" keeps the last occurrence.

  • Integer values keep the specified positional match.

Return type:

DataFrame

Returns:

tuple of pandas.Series – A tuple of pandas Series containing all original input data with the duplicates removed.

marine_qc.duplicate_checker.duplicates.build_dataframe(station_id, lat, lon, date, vsi, dsi, extra=None)[source]

Build a pandas DataFrame from the supplied columns.

Parameters:

  • station_id (SequenceStrType, optional) – One-dimensional array of station IDs.

  • lat (SequenceNumberType, optional) – One-dimensional array of latitudes in degrees.

  • lon (SequenceNumberType, optional) – One-dimensional array of longitudes in degrees.

  • date (SequenceDatetimeType, optional) – One-dimensional array of datetime values.

  • vsi (SequenceNumberType, optional) – One-dimensional reported speed array in km/h.

  • dsi (SequenceNumberType, optional) – One-dimensional reported heading array in degrees.

  • extra (dict, optional) – Additional column-value pairs.

Return type:

DataFrame

Returns:

pandas.DataFrame – A pandas DataFrame from the supplied columns.

marine_qc.duplicate_checker.duplicates.group_matches(matches, order_map)[source]

Re-order and group matched entity pairs according to a supplied ordering.

Parameters:

  • matches (pandas.DataFrame) – A pandas DataFrame that must contain the columns unique_id_l and unique_id_r. Each row represents a candidate match between a left-hand identifier (unique_id_l) and a right-hand identifier (unique_id_r).

  • order_map (dict[Any, Any]) – Mapping from an identifier to a sortable rank (e.g. integer). The function uses this map to:

    • decide whether to swap the left/right columns so that the left side always has the smaller rank,

    • sort the matches first by the left-hand rank and then by the right-hand rank.

Return type:

list[list[Any]]

Returns:

list[list[Any]] – A list of groups, where each group is a list of identifiers that are transitively connected through the matches. For example, if the input contains pairs (A, B) and (B, C), the result will contain a single group [A, B, C]. Unconnected pairs appear as separate two-element groups.

marine_qc.duplicate_checker.duplicates.make_comparison(column, compare_level_libraries, offsets, ignore_entries, ignore_nan_both, ignore_nan_either)[source]

Build a cl.CustomComparison for column.

Parameters:

  • column (str) – Name of the data column.

  • compare_level_libraries (dict) – Override comparison levels for selected columns. This modifies the comparison level used during comparison.

  • offsets (dict) – Override comparison offsets for selected columns. This modifies the tolerance used during comparison.

  • ignore_entries (dict) – Ignore specific values for selected columns during comparison.

  • ignore_nan_both (list of str) – For selected columns, consider two observations as duplicates if both values being compared are NaN.

  • ignore_nan_either (list of str) – For selected columns, consider two observations as duplicates if either value being compared is NaN.

Return type:

CustomComparison

Returns:

cl.CustomComparison – A splink comparison library instance.

marine_qc.duplicate_checker.duplicates.prepare_dataframe(data)[source]

Prepare a pandas DataFrame for detecting duplicates.

Parameters:

data (pandas.DataFrame) – A pandas DataFrame that should be prepared for detecting duplicates.

Return type:

DataFrame

Returns:

pandas.DataFrame – A prepared pandas DataFrame for detecting duplicates.

marine_qc.duplicate_checker.duplicates.prepare_nan_handling(nan_handling, columns)[source]

Resolve which DataFrame columns should be considered when handling NaN values or duplicate detection.

Parameters:

  • nan_handling (str, list of str, bool or None) – Specifies how NaN values are treated for duplicate detection on the selected columns.

    • True - treat all columns in columns as NaN-sensitive.

    • False - do not apply any NaN-handling (return an empty list).

    • None - do not apply any NaN-handling (return an empty list).

    • str - a single column name to which the NaN rule should be applied.

    • list[str] - an explicit list of column names to which the NaN rule should be applied.

  • columns (pandas.Index) – The complete index of column names present in the DataFrame.

Return type:

list[str]

Returns:

list of str – A list of column names that should be used for NaN-aware duplicate comparison, based on the value of nan_handling:

  • If nan_handling is True, the function returns a list of all column names.

  • If nan_handling is a string, it returns a one-element list containing that column.

  • If nan_handling is an empty list, False or None, it returns an empty list (no NaN handling).

  • If nan_handling is already a list of strings, it returns that list unchanged.

marine_qc.duplicate_checker.duplicates.reindex_nulls(df, null_label)[source]

Reindex a DataFrame in ascending order based on the number of ‘null’ strings in each row.

Parameters:

  • df (pandas.DataFrame) – Input DataFrame. Cells with the string “null” are counted as nulls.

  • null_label (Any) – Missing value representative.

Return type:

DataFrame

Returns:

pandas.DataFrame – DataFrame reindexed so that rows with fewer ‘null’ values appear first. Original row order is preserved for rows with the same null count.