{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cd5ea363-0f9b-480a-a6e4-d28338c442bb",
      "metadata": {},
      "outputs": [],
      "source": [
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b5d08168-f04f-4769-93f7-c5e3c25e5c0f",
      "metadata": {},
      "outputs": [],
      "source": [
        "from marine_qc import (\n",
        "    combine_qc_results,\n",
        "    do_climatology_check,\n",
        "    do_datetime_check,\n",
        "    do_hard_limit_check,\n",
        "    do_maritime_check,\n",
        "    do_multiple_individual_check,\n",
        "    do_position_check,\n",
        "    do_valid_value_check,\n",
        "    do_wind_consistency_check,\n",
        "    plot_latitude_longitude,\n",
        ")\n",
        "from marine_qc.helpers.external_clim import get_climatological_value"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3fa6ca48-7f38-4127-93e8-8a8f4119d5e7",
      "metadata": {},
      "outputs": [],
      "source": [
        "from data import get_climatology_data, get_individual_data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "56c5d833-c51f-4b59-8c75-9166cbcbf530",
      "metadata": {},
      "source": [
        "# How to use quality control checks with individual reports"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5cddbfc6-f401-4c8f-bfea-b08e09ed38ee",
      "metadata": {},
      "source": [
        "The ``marine_qc`` toolbox provides some quality control (QC) checks that work on individual marine reports. Creating some test dataset we can show how to use them on \"real\" data. There are several checks that are shown here:\n",
        "\n",
        "* a position check that tests whether the latitude-longitude position of the report is valid\n",
        "* a datetime check that tests whether both the date and the time of the report is valid\n",
        "* a valid-value check that tests whether an observational value is present\n",
        "* a hard-limit check that tests whether an observational value is within a specified range\n",
        "* a wind-consistency check that tests whether wind speed and wind direction are consistent\n",
        "* a maritime check that tests whether latitude-longitude is on sea whereby an external land-sea mask is needed\n",
        "* a climatology check that tests whether an observational value is within an acceptable range around the mean of an external climatology\n",
        "\n",
        "Finally, we run all these QC checks within a single function and combine the results of each QC check into a single QC flag.\n",
        "\n",
        "For more information of all available QC checks on individual reports see [the overview of QC functions for individual reports](https://marine-qc.readthedocs.io/en/stable/overview_ind.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b5fbca91-9de2-46f7-ae00-ef25c79dbefa",
      "metadata": {},
      "source": [
        "The test dataset we provide is a ``pandas.DataFrame`` including several individual marine reports. The QC functions return a QC flag for each individual report. The flags are:\n",
        "\n",
        "* `0`: the check has passed\n",
        "* `1`: the check has failed\n",
        "* `2`: the check is untestable"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b65d2a1a-4113-4d16-b416-5548a90424de",
      "metadata": {},
      "outputs": [],
      "source": [
        "data = get_individual_data()\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d1aa24c4-b676-4116-b359-22993f3e4fc4",
      "metadata": {},
      "source": [
        "We have nine different marine reports that include six different parameters (`lat`, `lon`, `date`, `sea_surface_temperature`, `wind_speed` and `wind_direction`)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a507a530-0a62-44b9-ae68-3a5311c7e7f3",
      "metadata": {},
      "source": [
        "## Checks that need no external data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "99e5b64c-e50d-48ad-831b-8602223305fd",
      "metadata": {},
      "source": [
        "We start with the position check. The function needs latitude and longitude values as input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b824a093-bf75-4d75-8d03-cf04b5bdc6da",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_pos = do_position_check(\n",
        "    lat=data.lat,\n",
        "    lon=data.lon,\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"lat\": data.lat, \"lon\": data.lon, \"qc_pos\": qc_pos})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "24fa0a8f-72cd-417c-af5b-1f8d23eec396",
      "metadata": {},
      "source": [
        "All checks have passed except the check for the South Pacific Ocean. It seems that latitude and longitude values are swapped. These are the expected results."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "755c038e-8f5e-4176-9138-4d1c2931c9bb",
      "metadata": {},
      "source": [
        "The next check is the datetime check which only needs date values as input."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b5a83d98-f81c-4bf9-bc06-77c7406e6454",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_datetime = do_datetime_check(\n",
        "    data.date,\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"date\": data.date, \"qc_datetime\": qc_datetime})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "440657a1-7825-4820-a1f1-6838f4ff0129",
      "metadata": {},
      "source": [
        "As expected all checks have passed except the check for the Gulf of Mexico. We get a `2` that stands for untestable since the datetime is not wrong but not available."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "84089b25-a178-4091-ab1c-fad80c3176c7",
      "metadata": {},
      "source": [
        "In the next few checks we focus on `sea_surface_temperature`. First of all, we want to check if the values are valid."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "24c95815-cf9f-4d63-b1cd-5bbb58576fa3",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_valid = do_valid_value_check(\n",
        "    data.sea_surface_temperature,\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"sea_surface_temperature\": data.sea_surface_temperature, \"qc_valid\": qc_valid})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cf541048-ee53-4c2d-8062-7632749c95e4",
      "metadata": {},
      "source": [
        "Since no values for Paris, Tokyo and Sydney are provided, the checks fail for these reports as expected."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "38d5173b-0257-433a-8405-11245615702b",
      "metadata": {},
      "source": [
        "Next, we check whether the values are within a range between `10` and `30` `°C`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d4e8fe82-6931-48e1-ba28-238c28bf1dc0",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_hard = do_hard_limit_check(\n",
        "    data.sea_surface_temperature,\n",
        "    limits=(10, 30),\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"sea_surface_temperature\": data.sea_surface_temperature, \"qc_hard\": qc_hard})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "15848f37-4c85-4644-bc4e-94b0a5a4a1bc",
      "metadata": {},
      "source": [
        "As expected, the check fails for the Norwegian Sea since the value is under `10 °C` (`8.5 °C`). Interstingly, we get three `2`s (untestable) for locations where no values are provided. The aim of the hard-limit test is not to test whether a value is valid but to test whether a value is within a specified range. Therefore, we decided to return `2`s instead of `1`s (failed)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "29ca9852-b7f4-48e0-9abd-e195f06f138b",
      "metadata": {},
      "source": [
        "The last check that only need inputs from the dataset is the wind-consistency check. This is a so-called cross check that test whether two observsational values are consistent. In this check, `wind_speed` and `wind_direction` are consistent if zero `wind_speed` corresponds to no particular `wind_direction` and `wind_speed` above a threshold corresponds to a particular `wind_direction`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8dd94224-347f-486c-a2ab-785442a09a4d",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_wind = do_wind_consistency_check(\n",
        "    wind_speed=data.wind_speed,\n",
        "    wind_direction=data.wind_direction,\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"wind_speed\": data.wind_speed, \"wind_direction\": data.wind_direction, \"qc_wind\": qc_wind})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d167ac75-cf3f-4e12-8e20-ba4f07b54585",
      "metadata": {},
      "source": [
        "All checks have passed except the check for the Norwegian Sea. Here, zero `wind_speed` correspond to a particular `wind_direction` (`270°`)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f623da5e-48f1-442a-8b41-ac1d37f48db3",
      "metadata": {},
      "source": [
        "## Checks that do need external data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "65894aad-b4f7-46c5-a2f4-ff8a6b58be66",
      "metadata": {},
      "source": [
        "Now, we focus on QC checks that need external data in addition to the individual reports. Firtly, we import some external data. This is a `xarray.Dataset` containing [CF](https://cfconventions.org/)-compliant data:\n",
        "\n",
        "* a land-sea mask where `1` denotes a land point and `0` denotes a sea point (`land_sea_mask`)\n",
        "* a simple, latitudinally and longitudinally dependent sea-surface temperature field (`sst`)\n",
        "* a sea-surface temperature standard deviation of grid cell minus neighbourhood mean (`sst_stde1`)\n",
        "* a sea-surface temperature standard deviation of point observation minus grid cell mean (`sst_stde2`)\n",
        "* a sea-surface temperature standard deviation of neighbourhood mean uncertainty (`sst_stde3`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0312239f-d623-4c54-a1a4-0284ae455ef4",
      "metadata": {},
      "outputs": [],
      "source": [
        "climatology_data = get_climatology_data()\n",
        "climatology_data"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c94f6ab0-1175-4a51-94bf-7e101169e258",
      "metadata": {},
      "source": [
        "Firstly, we check whether the positions of the marine reports are on sea using the external land-sea mask."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d0358795-7703-413f-a282-904d6f962f5b",
      "metadata": {},
      "outputs": [],
      "source": [
        "climatology_data.land_sea_mask.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2039d666-6c9e-44f8-9dd4-12eca1d81e7c",
      "metadata": {},
      "source": [
        "We do not need to extract the latitude-longitude points amnually from the land-sea mask. This is done internally in the maritime check. We simply can pass the entire land-sea mask as a `xarray.DataArray` to the function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "70752b0d-9db2-4a59-84ec-5630bf330b05",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_sea = do_maritime_check(\n",
        "    lat=data.lat,\n",
        "    lon=data.lon,\n",
        "    sea_land_mask=climatology_data.land_sea_mask,\n",
        "    sea_flag=0,\n",
        ")\n",
        "pd.DataFrame({\"location\": data.location, \"lat\": data.lat, \"lon\": data.lon, \"qc_sea\": qc_sea})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "46a30fe1-be1f-48d1-8120-e3df432f84d2",
      "metadata": {},
      "source": [
        "As expected all checks have passed except for the land points (Paris, Tokyo and Sydney)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "cb5a7568-ef32-4d71-bf7b-5e7a2d0c596e",
      "metadata": {},
      "source": [
        "Finally, we check `sea_surface_temeprature` against the sea-surface climatology. The check should pass if the value from the report and the climatological value do not differ more than `5K`. As in the check above, we can provide the external data as `xarray.DataArray`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b683ead1-aa5f-4230-a8e2-4877186fb4e5",
      "metadata": {},
      "outputs": [],
      "source": [
        "climatology_data.sst.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fd295612-3469-4eb6-9266-5caf6ad4ede8",
      "metadata": {},
      "source": [
        "We want to extract and show the climatological values that correspond to the latitude-longitude positions in the marine reports. Therefore, we can use the use the helper function `get_climatological_value` from `marine_qc.helpers.external_clim`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b50af497-4fdc-4011-b2f6-8c916449ae7c",
      "metadata": {},
      "outputs": [],
      "source": [
        "clim_sst = get_climatological_value(climatology_data.sst, lat=data.lat, lon=data.lon)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5c8e6f71-6c05-4960-be59-392dba37a0a0",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_clim = do_climatology_check(\n",
        "    value=data.sea_surface_temperature,\n",
        "    climatology=climatology_data.sst,\n",
        "    maximum_anomaly=5.0,\n",
        "    lat=data.lat,\n",
        "    lon=data.lon,\n",
        ")\n",
        "pd.DataFrame(\n",
        "    {\n",
        "        \"location\": data.location,\n",
        "        \"lat\": data.lat,\n",
        "        \"lon\": data.lon,\n",
        "        \"sea_surface_temperature\": data.sea_surface_temperature,\n",
        "        \"climatology\": clim_sst,\n",
        "        \"qc_clim\": qc_clim,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4001b066-ed3f-4f8e-8972-7b0d777781c8",
      "metadata": {},
      "source": [
        "As for the hard-limit test, we get `2`s (untestable) for reports without sea-surface temperatures. The check for the Gulf of Mexico fails the the temperature difference is more than `5K`. All other checks have passed as expected."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21e84ba5-3494-488d-89fd-7715d49030aa",
      "metadata": {},
      "source": [
        "## Run all checks within one function"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "129e4a82-0dc9-4873-84c8-e84afde0c809",
      "metadata": {},
      "source": [
        "We can run multiple QC checks within one function (`do_multiple_individual_check`). Therefore, we need a `pandas.DataFrame` and a nested QC dictionary. This is the structure of the dictionary:\n",
        "\n",
        "* arbitrary user-specified name for the checks\n",
        "  * \"func\": name of the QC function as `str` (mandatory)\n",
        "  * \"names\": dictionary containing parameter names of the QC function and their corresponding columns in the `pandas.DataFrame` (mandatory)\n",
        "  * \"arguments\": dictionary containing any key-word arguments passed to the QC function (optionally)\n",
        "* ...\n",
        "\n",
        "We define the QC dictionary according to the QC checks above."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "177319b1-e484-4359-8c57-bcd26fec81d3",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_dict = {\n",
        "    \"positional_check\": {\n",
        "        \"func\": \"do_position_check\",\n",
        "        \"names\": {\n",
        "            \"lat\": \"lat\",\n",
        "            \"lon\": \"lon\",\n",
        "        },\n",
        "    },\n",
        "    \"datetime_check\": {\n",
        "        \"func\": \"do_datetime_check\",\n",
        "        \"names\": {\"date\": \"date\"},\n",
        "    },\n",
        "    \"hard_limit_check\": {\"func\": \"do_hard_limit_check\", \"names\": {\"value\": \"sea_surface_temperature\"}, \"arguments\": {\"limits\": (10, 29)}},\n",
        "    \"maritime_check\": {\n",
        "        \"func\": \"do_maritime_check\",\n",
        "        \"names\": {\n",
        "            \"lat\": \"lat\",\n",
        "            \"lon\": \"lon\",\n",
        "        },\n",
        "        \"arguments\": {\n",
        "            \"sea_land_mask\": climatology_data.land_sea_mask,\n",
        "            \"sea_flag\": 0,\n",
        "        },\n",
        "    },\n",
        "    \"climatology_check\": {\n",
        "        \"func\": \"do_climatology_check\",\n",
        "        \"names\": {\"value\": \"sea_surface_temperature\"},\n",
        "        \"arguments\": {\n",
        "            \"climatology\": climatology_data.sst,\n",
        "            \"maximum_anomaly\": 5.0,\n",
        "            \"lat\": data.lat,\n",
        "            \"lon\": data.lon,\n",
        "        },\n",
        "    },\n",
        "    \"wind_consistency_check\": {\n",
        "        \"func\": \"do_wind_consistency_check\",\n",
        "        \"names\": {\"wind_speed\": \"wind_speed\", \"wind_direction\": \"wind_direction\"},\n",
        "    },\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "12254dfc-01c6-41a3-b5fa-217f58e34815",
      "metadata": {},
      "source": [
        "Now, run all QC checks calling one function. We set `return_method` to \"failed\" which means that all requested QC check are run until the first check fails. The other QC checks are flagged as unstested (`3`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0a8f04e3-fa76-4e15-9dbf-68d8fa77674b",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_multi = do_multiple_individual_check(\n",
        "    data,\n",
        "    qc_dict,\n",
        "    return_method=\"failed\",\n",
        ")\n",
        "qc_multi"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f53f3f09-8045-4473-a0bf-7cb5f00dc4f7",
      "metadata": {},
      "source": [
        "Now, we combine the results into one final QC flag. The QC flag values are prioritized in this order: [`1`, `0`, `3`, `2`]."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "be66324a-ef15-4148-b0d0-a17e856d2d95",
      "metadata": {},
      "outputs": [],
      "source": [
        "qc_flag = combine_qc_results(qc_multi)\n",
        "pd.DataFrame({**data, \"qc_flag\": qc_flag})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e6aa36a3-021b-4e73-8f37-ce1cdeb1b985",
      "metadata": {},
      "source": [
        "This is our final QC results. Let's make some plots."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "33106006-0219-4a39-949b-8b5274c6251c",
      "metadata": {},
      "outputs": [],
      "source": [
        "plot = plot_latitude_longitude(data.lat, data.lon, qc_flag, marker_size=15, add_coastlines=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "92b773ca-83db-4043-851e-6e7e1ce477e6",
      "metadata": {},
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.13.2"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}