Duplicate checker classes and functions

marine_qc.duplicate_checker.duplicates.reindex_nulls(df, null_label)[source]

Reindex a DataFrame in ascending order based on the number of ‘null’ strings in each row.

Parameters:

  • df (pandas.DataFrame) – Input DataFrame. Cells with the string “null” are counted as nulls.

  • null_label (Any) – Missing value representative.

Return type:

DataFrame

Returns:

pandas.DataFrame – DataFrame reindexed so that rows with fewer ‘null’ values appear first. Original row order is preserved for rows with the same null count.

class marine_qc.duplicate_checker.duplicates.DupDetect(groups, settings, data)[source]

Class to detect, flag, and remove duplicate entries in a DataFrame using a comparison matrix from splink.

Parameters:

  • groups (list of list of Any) – Groups of index pairs of potentially duplicated observations.

  • settings (dict) – Settings dict used for duplicate detection.

  • data (pandas.DataFrame) – Original dataset.

flag_duplicates(keep='first')[source]

Get result dataset with flagged duplicates.

Parameters:

keep (str or int, default: first) – Determines which duplicate entry should be retained.

  • "first" keeps the first occurrence.

  • "last" keeps the last occurrence.

  • Integer values keep the specified positional match.

Return type:

Series

Returns:

pandas.Series – Series containing duplicate flags for the detected observations.

References

get_duplicates(keep='first', overwrite=True)[source]

Identify duplicate matches based on the comparison matrix.

Parameters:

  • keep (str or int, default: first) – Determines which duplicate entry should be retained.

    • "first" keeps the first occurrence.

    • "last" keeps the last occurrence.

    • Integer values keep the specified positional match.

  • overwrite (bool, default: True) – Whether to recompute matches if already calculated.

Return type:

Series

Returns:

pandas.Series – A pandas Series containing the indexes of the corresponding duplicate(s).

remove_duplicates(keep='first')[source]

Remove duplicate entries from the dataset.

Parameters:

keep (str or int) – Determines which duplicate entry should be retained.

  • "first" keeps the first occurrence.

  • "last" keeps the last occurrence.

  • Integer values keep the specified positional match.

Return type:

DataFrame

Returns:

tuple of pandas.Series – A tuple of pandas Series containing all original input data with the duplicates removed.