{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "36f53233-8158-454b-b0b6-9904f5250cf8",
      "metadata": {},
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e73a7b60-c93d-466a-a5bd-390cd3701ef9",
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1c95b7bf-0f23-4558-b411-2bac6f753a01",
      "metadata": {},
      "outputs": [],
      "source": [
        "from marine_qc import (\n",
        "    combine_qc_results,\n",
        "    do_iquam_track_check,\n",
        "    do_multiple_sequential_check,\n",
        "    do_spike_check,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e932294d-d925-4a69-9b89-5dea456a9a13",
      "metadata": {},
      "outputs": [],
      "source": [
        "from data import get_sequential_data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0f6009ea-ba4e-4098-82df-657958aa806a",
      "metadata": {},
      "source": [
        "# How to use quality control checks with sequential reports from ships"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "74f5740d-a913-4179-965b-93911e8338eb",
      "metadata": {},
      "source": [
        "The ``marine_qc`` toolbox provides some quality control (QC) checks that work on sequential marine ship reports. Creating some test dataset we can show how to use them on \"real\" data. There are two checks that are shown here:\n",
        "\n",
        "* a spike check that tests that a sequence of values has unlikely “spikes” in it.\n",
        "* a track check that tests that the locations of a series of reports form a plausible ship track\n",
        "\n",
        "Finally, we run all these QC checks within a single function and combine the results of each QC check into a single QC flag.\n",
        "\n",
        "For more information of all available QC checks on sequential ship reports see [the overview of QC functions for sequential ship reports](https://marine-qc.readthedocs.io/en/stable/overview_seq.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5653375d-3640-41fc-8771-8f7092ca5d8c",
      "metadata": {},
      "source": [
        "The test dataset we provide is a ``pandas.DataFrame`` including sequential marine reports representing the track of one ship. The QC functions return a QC flag for each individual report. The flags are:\n",
        "\n",
        "* `0`: the check has passed\n",
        "* `1`: the check has failed\n",
        "* `2`: the check is untestable"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3244744c-acd5-4c1e-bd9c-cd6769721dbe",
      "metadata": {},
      "outputs": [],
      "source": [
        "data = get_sequential_data()\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "10de3720-8173-4c52-86aa-c22308eadcb2",
      "metadata": {},
      "source": [
        "The ship data include four different parameters (`lat`, `lon`, `date`, and `sst`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7c79b155-3c09-4a91-8e36-958badd63ffe",
      "metadata": {},
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(1, 2, figsize=(13, 5))\n",
        "\n",
        "axs[0].plot(data[\"lon\"], data[\"lat\"], marker=\"o\", linestyle=\"-\")\n",
        "axs[0].set_title(\"Ship – latitude/longitude track\")\n",
        "axs[0].set_xlabel(\"Longitude (°)\")\n",
        "axs[0].set_ylabel(\"Latitude (°)\")\n",
        "axs[0].grid(True)\n",
        "\n",
        "axs[1].plot(data[\"sst\"], marker=\"o\", linestyle=\"-\")\n",
        "axs[1].set_title(\"Ship – sea surface temperature\")\n",
        "axs[1].set_xlabel(\"Time\")\n",
        "axs[1].set_ylabel(\"Temperature (°C)\")\n",
        "axs[1].grid(True)\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "00abadb2-29b9-4cc1-99cd-e5a31c757d2f",
      "metadata": {},
      "source": [
        "Firstly, a spike check is performed. Beside `lat`, `lon`, `date` and `value`, this check needs some extra parameters:\n",
        "\n",
        "* `max_gradient_space`: Maximum allowed spatial gradient (`0.5`: \"0.5 K per kilometer\")\n",
        "* `max_gradient_time`: Maximum allowed temporal gradient (`1.0`: \"1.0 K per kilometer\")\n",
        "* `delta_t`: Temperature delta used in the comparison (`2.0`: \"2.0 K\")\n",
        "* `n_neighbours`: Number of neighboring points considered in the analysis (`5`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "55d0c7fe-9f31-4740-8a84-7b8eedbca766",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_spike = do_spike_check(\n",
        "    value=data.sst,\n",
        "    lat=data.lat,\n",
        "    lon=data.lon,\n",
        "    date=data.date,\n",
        "    max_gradient_space=0.5,\n",
        "    max_gradient_time=1.0,\n",
        "    delta_t=2.0,\n",
        "    n_neighbours=5,\n",
        ")\n",
        "pd.DataFrame({\"lat\": data.lat, \"lon\": data.lon, \"date\": data.date, \"sst\": data.sst, \"qc_spike\": qc_spike})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "afb0a8f3-032f-426b-a1fb-5cf5cd43e16b",
      "metadata": {},
      "source": [
        "In the plot above, we can see three obvious spikes that are represented in the results of the spike check (`1` (failed)). The spike checks gives the expected results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "868c0d7d-d6b5-48e2-b9e8-e31ba4c5e47a",
      "metadata": {},
      "source": [
        "Now, we do the track check. As for the spike check we need some extra parameters:\n",
        "\n",
        "* `speed_limit`: Speed limit of platform in kilometers per hour (`25.0`)\n",
        "* `delta_d`: Latitude tolerance in degrees (`1.11`)\n",
        "* `delta_t`: Time tolerance in hundredths of an hour (`0.01`)\n",
        "* `n_neighbours`: Number of neighbouring points considered in the analysis (`5`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5e0ceb82-b57a-4b10-9f77-f36b5f8904e5",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_track = do_iquam_track_check(\n",
        "    lat=data.lat,\n",
        "    lon=data.lon,\n",
        "    date=data.date,\n",
        "    speed_limit=25.0,\n",
        "    delta_d=1.11,\n",
        "    delta_t=0.01,\n",
        "    n_neighbours=5,\n",
        ")\n",
        "pd.DataFrame({\"lat\": data.lat, \"lon\": data.lon, \"date\": data.date, \"qc_track\": qc_track})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "456e23fd-7faf-4a27-9598-51de02355b45",
      "metadata": {},
      "source": [
        "Within the given parameters, the ship is too \"fast\" between (44.387443, -28.748580) and (43.730756, -27.658971)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6b158926-aca3-4e30-9b30-2ff6eb1d926c",
      "metadata": {},
      "source": [
        "Now, we can run these two QC checks within a single function `do_multiple_sequential_check`. Therefore, we need a `pandas.DataFrame` and a nested QC dictionary. This is the structure of the dictionary:\n",
        "\n",
        "* arbitrary user-specified name for the checks\n",
        "  * \"func\": name of the QC function as `str` (mandatory)\n",
        "  * \"names\": dictionary containing parameter names of the QC function and their corresponding columns in the `pandas.DataFrame` (mandatory)\n",
        "  * \"arguments\": dictionary containing any key-word arguments passed to the QC function (optionally)\n",
        "* ...\n",
        "\n",
        "We define the QC dictionary according to the QC checks above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85082f92-0edd-45d1-80fb-378f3aa9ce65",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_dict = {\n",
        "    \"spike_check\": {\n",
        "        \"func\": \"do_spike_check\",\n",
        "        \"names\": {\n",
        "            \"value\": \"sst\",\n",
        "            \"lat\": \"lat\",\n",
        "            \"lon\": \"lon\",\n",
        "            \"date\": \"date\",\n",
        "        },\n",
        "        \"arguments\": {\n",
        "            \"max_gradient_space\": 0.5,\n",
        "            \"max_gradient_time\": 1.0,\n",
        "            \"delta_t\": 1.0,\n",
        "            \"n_neighbours\": 5,\n",
        "        },\n",
        "    },\n",
        "    \"iquam_track_check\": {\n",
        "        \"func\": \"do_iquam_track_check\",\n",
        "        \"names\": {\n",
        "            \"lat\": \"lat\",\n",
        "            \"lon\": \"lon\",\n",
        "            \"date\": \"date\",\n",
        "        },\n",
        "        \"arguments\": {\n",
        "            \"speed_limit\": 25.0,\n",
        "            \"delta_d\": 1.11,\n",
        "            \"delta_t\": 0.01,\n",
        "            \"n_neighbours\": 5,\n",
        "        },\n",
        "    },\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "145bcceb-a3f9-44e5-9393-23ea5b3513a1",
      "metadata": {},
      "source": [
        "We set `return_method` to \"failed\" which means that all requested QC check are run until the first check fails. The other QC checks are flagged as unstested (`3`). Furthermore, we set `groupby` to \"ship_id\" which specifies how the data should be grouped before applying QC functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ba42ca2a-66cd-4573-93d2-f6e9848969e3",
      "metadata": {
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "qc_multi = do_multiple_sequential_check(\n",
        "    data,\n",
        "    qc_dict=qc_dict,\n",
        "    groupby=\"ship_id\",\n",
        "    return_method=\"failed\",\n",
        ")\n",
        "qc_multi"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1a474e95-8a13-421f-8547-1ba4e6e84475",
      "metadata": {},
      "source": [
        "Now, we combine the results into one final QC flag. The QC flag values are prioritized in this order: [1, 0, 3, 2]."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "73f636de-230f-4711-9abf-666985f2573b",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_flag = combine_qc_results(qc_multi)\n",
        "pd.DataFrame({**data, \"qc_flag\": qc_flag})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "549b357a-2b4d-4794-9f55-57c0f343c511",
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.13.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}