{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5fbc1dee",
   "metadata": {},
   "source": [
    "# July 2021 floods: starter notebook\n",
    "\n",
    "This notebook does two things:\n",
    "1. **Connects** you to the challenge data in the cloud. Nothing is downloaded, and no login is needed: the data is public.\n",
    "2. **Shows what data is available** (EMS flood maps, EFAS flood hazard, GHS-OBAT buildings, GHS-POP population for DE, BE, NL).\n",
    "\n",
    "Click **Run all**. Works in Colab and on a laptop.\n",
    "\n",
    "<details>\n",
    "<summary><b>Not using Colab? (VS Code, Jupyter, any Python)</b></summary>\n",
    "\n",
    "1. `pip install gcsfs pyogrio rasterio geopandas shapely ipykernel`\n",
    "   (on Windows, if that fails: `conda install -c conda-forge geopandas rasterio pyogrio gcsfs`)\n",
    "2. VS Code: install the Python and Jupyter extensions and pick your environment as kernel. If the packages are already installed there, you can skip the `pip install` cell below. After installing, restart the kernel.\n",
    "\n",
    "**Problems:** *connection or timeout errors* → check your internet connection and retry · *`No such file` on a path* → run the Explore cell to see the exact file names · *`ModuleNotFoundError`* → the kernel and your terminal use different Pythons: install with `%pip install gcsfs` inside the notebook, then restart the kernel.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa4f5c8",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1a568b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "%pip install -q gcsfs pyogrio rasterio geopandas shapely"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a6951de",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, re, warnings\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "BUCKET = \"datasets_challenge\"\n",
    "FOLDER = \"working_data\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e4db53a",
   "metadata": {},
   "source": [
    "## 2. Connect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96bc9f92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gcsfs\n",
    "\n",
    "# The bucket is public: no login needed.\n",
    "os.environ[\"GS_NO_SIGN_REQUEST\"] = \"YES\"      # lets GDAL (rasterio, pyogrio) read it anonymously\n",
    "GDAL_PREFIX = \"/vsigs/\"\n",
    "ROOT = f\"{BUCKET}/{FOLDER}\"\n",
    "fs = gcsfs.GCSFileSystem(token=\"anon\")\n",
    "print(\"Connected:\", [Path(p).name for p in fs.ls(ROOT)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce75ad49",
   "metadata": {},
   "source": [
    "## 3. Explore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b82f013",
   "metadata": {},
   "outputs": [],
   "source": [
    "for folder in [\"cems-efas\", \"ghs_pop\", \"obat_be\", \"obat_de\", \"obat_nl\"]:\n",
    "    print(f\"\\n{folder}\")\n",
    "    for p in sorted(fs.find(f\"{ROOT}/{folder}\")):\n",
    "        if p.lower().endswith((\".tif\", \".tiff\", \".gpkg\")): print(\"  \", Path(p).name, f\"{fs.size(p)/1e6:,.0f} MB\")\n",
    "\n",
    "# EMS: one row per area (aoi), type (DEL/GRA) and version, with its layers\n",
    "EMS_RE = re.compile(r\"(EMSR\\d+)_(AOI\\d+)_([A-Z]+)_(PRODUCT|MONIT\\d*)_(\\w+?)_r\\d+_v\\d+\", re.I)\n",
    "rows = []\n",
    "for cc in (\"de\", \"be\", \"nl\"):\n",
    "    for p in fs.find(f\"{ROOT}/ems_{cc}\"):\n",
    "        m = EMS_RE.search(Path(p).name) if p.lower().endswith(\".shp\") else None\n",
    "        if m: rows.append(dict(cc=cc, aoi=m[2], ptype=m[3].upper(), ver=m[4].upper(), layer=m[5]))\n",
    "display(pd.DataFrame(rows).groupby([\"cc\", \"aoi\", \"ptype\", \"ver\"]).layer.apply(lambda s: \", \".join(sorted(s))).to_frame())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c726ab00",
   "metadata": {},
   "source": [
    "## 4. Your analysis\n",
    "Read data straight from the bucket, no login needed: list files with `fs` (gcsfs), or open them with GDAL-based libraries using the `GDAL_PREFIX` path prefix, for example `rasterio.open(f\"{GDAL_PREFIX}{path}\")`. The bucket is read-only, so save results to the runtime disk and download them from the file browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d08bb833",
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "starter_notebook_flood_challenge.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}