import pandas as pd

from marine_qc import (
    duplicate_check,
    flag_duplicates,
    get_duplicates,
    remove_duplicates,
)

from data import get_advanced_data

How to use the duplicate checker to detect possibly duplicated observations¶

The marine_qc toolbox provides some checks to detect possibly duplicated observations. Creating some test dataset we can show how to use them on “real” data. There are several checks that are shown here:

a duplicate check to detect possibly duplicated observations
a function that gets potentially duplicated observations and returns the corresponding detected duplicate matches for each observation
a function that flags potentially duplicated observations and returns the duplicate flags for each observation
a function that removes potentially duplicated observations and returns the input data with duplicates excluded

For more information of all available QC checks on individual reports see the overview of QC functions to detect possibly duplicated observations.

The test dataset we provide is a pandas.DataFrame including several individual marine reports.

data = get_advanced_data()
data

	station_id	lon	lat	date	vsi	dsi	sst
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0

We have several different marine reports that include six different parameters (lat, lon, date, sst (sea-surface temperature), vsi (ship speed), and dsi (ship heading)).

First of all, we detect possibly duplicated observations. This function returns a marine_qc.duplicate_checker.duplicates.DupDetect class that contains all information needed for the other duplicate functions.

Use the default settings of the duplicate checker¶

To introduce the duplicate check functions, we work with the default settings.

Note:

By default, the duplicate check comprises six kinds of observational values:

station_id: Any kind of idnetifier of the observing station (e.g. ship name, WIGOS Station Identifier).

lat: Latitude of the observing station in degrees.

lon: Longitude of the observing station in degrees.

date: Time stamp of the obersing station (datetime object or datetime-formatted string).

vsi: Speed of the observing station in kilometers per hour.

dsi: Course of the observing station in degrees.

By default, duplicate observations are detected according to the following criteria:

station_id: values must match exactly for observations to be considered duplicates.

lat: values may differ by up to 0.11 degrees for observations to be considered duplicates.

lon: values may differ by up to 0.11 degrees for observations to be considered duplicates.

date: values may differ by up to 1 minute for observations to be considered duplicates.

vsi: values may differ by up to 0.09 kilometers per hour for observations to be considered duplicates.

dsi: values may differ by up to 0.9 degrees for observations to be considered duplicates.

Later, we will see how to add extra kinds of observational values and their corresponding criteria.

Firstly, we run the duplicate_check. The result can be used to run the other duplicate check functions. Alternatively, the functions can be run directly with the data itself. If so, duplicate_check is run internally.

detected = duplicate_check(data=data)

dups = get_duplicates(
    detected=detected,
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	H
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, N]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

As we can see, the following duplicates has been detected:

data.loc[["B", "H"]]

	station_id	lon	lat	date	vsi	dsi	sst
B	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	315.0	302.0
H	AA	29.7	71.3	2022-02-01 00:00:59	4.11552	315.0	300.0

We expect these to be detected as duplicates since:

station_id: do match exactly
lon: do match exactly
lat: do match exactly
date: differ less than 1 minute
vsi: do match exactly
dsi: do match exactly

data.loc[["C", "J"]]

	station_id	lon	lat	date	vsi	dsi	sst
C	AA	32.0	71.2	2022-02-01	NaN	NaN	300.0
J	AA	32.0	71.2	2022-02-01	NaN	NaN	300.0

We expect these to be detected as duplicates since:

station_id: do match exactly
lon: do match exactly
lat: do match exactly
date: do match exactly
vsi: are both NaN
dsi: are both NaN

data.loc[["D", "L", "N"]]

	station_id	lon	lat	date	sst
D	AA	8.10	66.00	2022-02-01	300.0
L	AA	8.15	66.05	2022-02-01	300.0
N	AA	8.05	65.95	2022-02-01	300.0

We expect these to be detected as duplicates since:

station_id: do match exactly
lon: differ less than 0.11 degrees
lat: differ less than 0.11 degrees
date: do match exactly
vsi: do match exactly
dsi: do match exactly

data.loc[["E", "F", "Q"]]

	station_id	lon	lat	date	sst
E	AA	-21.2	65.8	2022-02-01	300.0
F	AA	-21.2	65.8	2022-02-01	300.0
Q	AA	-21.1	65.9	2022-02-01	300.0

We expect these to be detected as duplicates since:

station_id: do match exactly
lon: differ less than 0.11 degrees
lat: differ less than 0.11 degrees
date: do match exactly
vsi: do match exactly
dsi: do match exactly

Now, we can flag the duplicates. The flagging scheme is:

0: unique observation
1: best duplicate
3: worst duplicate

flags = flag_duplicates(detected=detected)
pd.DataFrame({**data, "duplicate_flag": flags})

	station_id	lon	lat	date	vsi	dsi	sst	duplicate_flag
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	0
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	1
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	1
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	1
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	1
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	3
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	0
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	3
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	0
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	3
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	0
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	3
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	0
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	3
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	0
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	0
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	3
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	0
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	0
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	0
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	0

Optionally, we can remove the duplicated observations.

series = remove_duplicates(detected=detected)
pd.DataFrame(series).T

	station_id	lon	lat	date	vsi	dsi	sst
A	AA	22.3	71.3	2022-02-01 00:00:00	0.0	0.0	300.0
B	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	315.0	302.0
C	AA	32.0	71.2	2022-02-01 00:00:00	NaN	NaN	300.0
D	AA	8.1	66.0	2022-02-01 00:00:00	0.0	0.0	300.0
E	AA	-21.2	65.8	2022-02-01 00:00:00	0.0	0.0	300.0
G	AA	21.2	-65.8	2022-02-01 00:00:00	0.0	0.0	300.0
I	AA	29.7	71.3	2022-02-02 00:00:00	4.11552	315.0	300.0
K	AA	8.5	66.0	2022-02-01 00:00:00	0.0	0.0	300.0
M	AB	8.1	66.0	2022-02-01 00:00:00	0.0	0.0	300.0
O	BB	8.1	66.0	2022-02-01 00:00:00	0.0	0.0	300.0
P	AA	-21.4	65.6	2022-02-01 00:00:00	0.0	0.0	300.0
R	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	316.0	300.0
S	AA	29.7	71.3	2022-02-01 00:00:00	4.0	315.0	300.0
T	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	NaN	300.0
U	AA	29.7	71.3	2022-02-01 00:00:00	NaN	315.0	300.0

Manipulate the settings of the duplicate checker¶

Here are some more examples of how to use the duplicate checker with user-specific settings.

ignore_entires: Ignore specific values for selected columns during comparison.
ignore_nan_both: For selected columns, consider two observations as duplicates whenever both compared value are NaN. By default, this is true for all columns.
ignore_nan_either: For selected columns, consider two observations as duplicates whenever either compared value is NaN.
ignore_columns: Column names to exclude entirely from duplicate detection.
offsets: Override comparison offsets for selected columns.
compare_level_libraries: Override comparison levels for selected columns. For more information see Splink: Comparison Level Library.

`ignore_entries`¶

If station_id of an observation is “AA” or “BB”, this entry is considered placeholder which is compatible with and equivalent to any other station_id value.

dups = get_duplicates(
    data=data,
    ignore_entries={"station_id": ["AA", "BB"]},
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	H
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, M, N, O]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	D
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	D
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

In addition to the default example, we get the following duplicates:

data.loc[["D", "L", "M", "N", "O"]]

	station_id	lon	lat	date	sst
D	AA	8.10	66.00	2022-02-01	300.0
L	AA	8.15	66.05	2022-02-01	300.0
M	AB	8.10	66.00	2022-02-01	300.0
N	AA	8.05	65.95	2022-02-01	300.0
O	BB	8.10	66.00	2022-02-01	300.0

“O” is detected as duplicate as expected since we use “BB” as a placeholder.
“M” is detected as duplicate as expected since we use “AA” and “BB” as placeholders which makes “AB” the only valid station_id.

`ignore_nan_both`¶

Two observations are considered as duplicates whenever both compared value are NaN only for selected column vsi.

dups = get_duplicates(
    data=data,
    ignore_nan_both="vsi",
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	H
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	NaN
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, N]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	NaN
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

In comparison with the default example “C” and “J” are not considered as duplicates since dsi has NaN values:

data.loc[["C", "J"]]

	station_id	lon	lat	date	vsi	dsi	sst
C	AA	32.0	71.2	2022-02-01	NaN	NaN	300.0
J	AA	32.0	71.2	2022-02-01	NaN	NaN	300.0

`ignore_nan_either`¶

For vsi, consider two observations as duplicates whenever either compared value is NaN.

dups = get_duplicates(
    data=data,
    ignore_nan_either="vsi",
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	[H, U, S]
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, N]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	B
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	B

The duplicate group (“B”, “H”) is extended by (“U”, “S”).

data.loc[["B", "H", "U", "S"]]

	station_id	lon	lat	date	vsi	dsi	sst
B	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	315.0	302.0
H	AA	29.7	71.3	2022-02-01 00:00:59	4.11552	315.0	300.0
U	AA	29.7	71.3	2022-02-01 00:00:00	NaN	315.0	300.0
S	AA	29.7	71.3	2022-02-01 00:00:00	4.00000	315.0	300.0

Warning:

Here we can see, that ignore_nan_either can lead to misleading duplicate chains:

(“B”, “H”) is obviously a duplicate pair.
(“H”, “U”) is a duplicate pair as well since we ignore NaNs for vsi.
(“U”, “S”) is a duplicate pair as well since we ignore NaNs for vsi.

This leads to the following duplicate chain:

“B” -> “H” -> “U” -> “S”

But obviously, (“B”, “S”) is not a duplicate pair since vsi differs more than 0.11 degrees.

`ignore_columns`¶

We exclude station_id entirely from duplicate detection.

dups = get_duplicates(
    data=data,
    ignore_columns="station_id",
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	H
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, M, N, O]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	D
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	D
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

We extend one duplicate group from the default example since we ignore station_id completely.

data.loc[["D", "L", "M", "N", "O"]]

	station_id	lon	lat	date	sst
D	AA	8.10	66.00	2022-02-01	300.0
L	AA	8.15	66.05	2022-02-01	300.0
M	AB	8.10	66.00	2022-02-01	300.0
N	AA	8.05	65.95	2022-02-01	300.0
O	BB	8.10	66.00	2022-02-01	300.0

`offsets`¶

We now consider two observations as duplicates if:

lat differs up to 1.0 degrees
lon differs up to 1.0 degrees
date differs up to 360 seconds (1 hour)

dups = get_duplicates(
    data=data,
    offsets={"lat": 1.0, "lon": 1.0, "date": 360},
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	H
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[K, L, N]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, P, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	B
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	D
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	E
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

We expand two duplicate groups:

data.loc[["D", "K", "L", "N"]]

	station_id	lon	lat	date	sst
D	AA	8.10	66.00	2022-02-01	300.0
K	AA	8.50	66.00	2022-02-01	300.0
L	AA	8.15	66.05	2022-02-01	300.0
N	AA	8.05	65.95	2022-02-01	300.0

data.loc[["E", "F", "P", "Q"]]

	station_id	lon	lat	date	sst
E	AA	-21.2	65.8	2022-02-01	300.0
F	AA	-21.2	65.8	2022-02-01	300.0
P	AA	-21.4	65.6	2022-02-01	300.0
Q	AA	-21.1	65.9	2022-02-01	300.0

`compare_level_libraries`¶

Finally, we add sst with an offset of 1K to the criteria list.

dups = get_duplicates(
    data=data,
    compare_level_libraries={"sst": "AbsoluteDifferenceLevel"},
    offsets={"sst": 1.0},
)
pd.DataFrame({**data, "duplicates": dups})

	station_id	lon	lat	date	vsi	dsi	sst	duplicates
A	AA	22.30	71.30	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
B	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	315.0	302.0	NaN
C	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	J
D	AA	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	[L, N]
E	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	[F, Q]
F	AA	-21.20	65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	E
G	AA	21.20	-65.80	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
H	AA	29.70	71.30	2022-02-01 00:00:59	4.11552	315.0	300.0	NaN
I	AA	29.70	71.30	2022-02-02 00:00:00	4.11552	315.0	300.0	NaN
J	AA	32.00	71.20	2022-02-01 00:00:00	NaN	NaN	300.0	C
K	AA	8.50	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
L	AA	8.15	66.05	2022-02-01 00:00:00	0.00000	0.0	300.0	D
M	AB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
N	AA	8.05	65.95	2022-02-01 00:00:00	0.00000	0.0	300.0	D
O	BB	8.10	66.00	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
P	AA	-21.40	65.60	2022-02-01 00:00:00	0.00000	0.0	300.0	NaN
Q	AA	-21.10	65.90	2022-02-01 00:00:00	0.00000	0.0	300.0	E
R	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	316.0	300.0	NaN
S	AA	29.70	71.30	2022-02-01 00:00:00	4.00000	315.0	300.0	NaN
T	AA	29.70	71.30	2022-02-01 00:00:00	4.11552	NaN	300.0	NaN
U	AA	29.70	71.30	2022-02-01 00:00:00	NaN	315.0	300.0	NaN

Now (“B”, “H”) is no duplicate pair anymore since sst differs more than 1 Kelvin.

data.loc[["B", "H"]]

	station_id	lon	lat	date	vsi	dsi	sst
B	AA	29.7	71.3	2022-02-01 00:00:00	4.11552	315.0	302.0
H	AA	29.7	71.3	2022-02-01 00:00:59	4.11552	315.0	300.0

How to use the duplicate checker to detect possibly duplicated observations¶

Use the default settings of the duplicate checker¶

Manipulate the settings of the duplicate checker¶

ignore_entries¶

ignore_nan_both¶

ignore_nan_either¶

ignore_columns¶

offsets¶

compare_level_libraries¶

`ignore_entries`¶

`ignore_nan_both`¶

`ignore_nan_either`¶

`ignore_columns`¶

`offsets`¶

`compare_level_libraries`¶