1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
|
{
"cells": [
{
"cell_type": "markdown",
"id": "5fbc1dee",
"metadata": {},
"source": [
"# July 2021 floods: starter notebook\n",
"\n",
"This notebook does two things:\n",
"1. **Connects** you to the challenge data in the cloud. Nothing is downloaded, and no login is needed: the data is public.\n",
"2. **Shows what data is available** (EMS flood maps, EFAS flood hazard, GHS-OBAT buildings, GHS-POP population for DE, BE, NL).\n",
"\n",
"Click **Run all**. Works in Colab and on a laptop.\n",
"\n",
"<details>\n",
"<summary><b>Not using Colab? (VS Code, Jupyter, any Python)</b></summary>\n",
"\n",
"1. `pip install gcsfs pyogrio rasterio geopandas shapely ipykernel`\n",
" (on Windows, if that fails: `conda install -c conda-forge geopandas rasterio pyogrio gcsfs`)\n",
"2. VS Code: install the Python and Jupyter extensions and pick your environment as kernel. If the packages are already installed there, you can skip the `pip install` cell below. After installing, restart the kernel.\n",
"\n",
"**Problems:** *connection or timeout errors* → check your internet connection and retry · *`No such file` on a path* → run the Explore cell to see the exact file names · *`ModuleNotFoundError`* → the kernel and your terminal use different Pythons: install with `%pip install gcsfs` inside the notebook, then restart the kernel.\n",
"\n",
"</details>"
]
},
{
"cell_type": "markdown",
"id": "1fa4f5c8",
"metadata": {},
"source": [
"## 1. Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2f1a568b",
"metadata": {},
"outputs": [],
"source": [
"%%capture\n",
"%pip install -q gcsfs pyogrio rasterio geopandas shapely"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a6951de",
"metadata": {},
"outputs": [],
"source": [
"import os, re, warnings\n",
"from pathlib import Path\n",
"import pandas as pd\n",
"warnings.filterwarnings(\"ignore\")\n",
"\n",
"BUCKET = \"datasets_challenge\"\n",
"FOLDER = \"working_data\""
]
},
{
"cell_type": "markdown",
"id": "7e4db53a",
"metadata": {},
"source": [
"## 2. Connect"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "96bc9f92",
"metadata": {},
"outputs": [],
"source": [
"import gcsfs\n",
"\n",
"# The bucket is public: no login needed.\n",
"os.environ[\"GS_NO_SIGN_REQUEST\"] = \"YES\" # lets GDAL (rasterio, pyogrio) read it anonymously\n",
"GDAL_PREFIX = \"/vsigs/\"\n",
"ROOT = f\"{BUCKET}/{FOLDER}\"\n",
"fs = gcsfs.GCSFileSystem(token=\"anon\")\n",
"print(\"Connected:\", [Path(p).name for p in fs.ls(ROOT)])"
]
},
{
"cell_type": "markdown",
"id": "ce75ad49",
"metadata": {},
"source": [
"## 3. Explore"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b82f013",
"metadata": {},
"outputs": [],
"source": [
"for folder in [\"cems-efas\", \"ghs_pop\", \"obat_be\", \"obat_de\", \"obat_nl\"]:\n",
" print(f\"\\n{folder}\")\n",
" for p in sorted(fs.find(f\"{ROOT}/{folder}\")):\n",
" if p.lower().endswith((\".tif\", \".tiff\", \".gpkg\")): print(\" \", Path(p).name, f\"{fs.size(p)/1e6:,.0f} MB\")\n",
"\n",
"# EMS: one row per area (aoi), type (DEL/GRA) and version, with its layers\n",
"EMS_RE = re.compile(r\"(EMSR\\d+)_(AOI\\d+)_([A-Z]+)_(PRODUCT|MONIT\\d*)_(\\w+?)_r\\d+_v\\d+\", re.I)\n",
"rows = []\n",
"for cc in (\"de\", \"be\", \"nl\"):\n",
" for p in fs.find(f\"{ROOT}/ems_{cc}\"):\n",
" m = EMS_RE.search(Path(p).name) if p.lower().endswith(\".shp\") else None\n",
" if m: rows.append(dict(cc=cc, aoi=m[2], ptype=m[3].upper(), ver=m[4].upper(), layer=m[5]))\n",
"display(pd.DataFrame(rows).groupby([\"cc\", \"aoi\", \"ptype\", \"ver\"]).layer.apply(lambda s: \", \".join(sorted(s))).to_frame())"
]
},
{
"cell_type": "markdown",
"id": "c726ab00",
"metadata": {},
"source": [
"## 4. Your analysis\n",
"Read data straight from the bucket, no login needed: list files with `fs` (gcsfs), or open them with GDAL-based libraries using the `GDAL_PREFIX` path prefix, for example `rasterio.open(f\"{GDAL_PREFIX}{path}\")`. The bucket is read-only, so save results to the runtime disk and download them from the file browser."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d08bb833",
"metadata": {},
"outputs": [],
"source": [
"# your code"
]
}
],
"metadata": {
"colab": {
"name": "starter_notebook_flood_challenge.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|
