Eve library

@gaia/weather-data

Other

Revision 1

On this page

Sovereign weather & climate data foundation for Phase 175. Provides the ingestion clients, canonical sample schema, normalisation statistics, temporal splitting, and stratified curriculum sampling that feed every downstream model (GraphCast, GenCast, cyclone, nowcast, climate emulator, downscaling).

Modules#

Module Phase Contents
variables — Canonical 37-level ERA5 variable set + published GraphCast per-variable/per-level loss weights + flat state-vector layout.
schema 175.1.3.1 WeatherSample, lat/lon & icosahedral grid specs, blake2b provenance hashing, Zarr chunk alignment.
statistics 175.1.3.4 Numerically-stable (Chan parallel) per-variable/per-level mean/std/min/max, diff (tendency) stats, versioned save/load, exact normalise/denormalise.
splits 175.1.3.3 Strictly temporal train (≤2018) / val (2019–2020) / test (2021+) split with an embargo band that prevents rollout leakage across boundaries.
sampling 175.1.3.2 Extreme-event labelling + inverse-frequency stratified sampler + curriculum scheduler that up-weights cyclones/atmospheric-rivers/blocking over training.
ingest.reanalysis 175.1.1.1–5 ERA5, ERA5-Land, IFS HRES (MARS), GFS (NOMADS), MERRA-2 (GES DISC) request builders + MinIO hot/warm/cold tiering.
ingest.ibtracs 175.1.1.6 IBTrACS v04 best-track CSV parser → cyclone tracks with Saffir-Simpson classification and R34/R50/R64 wind radii.
ingest.nexrad 175.1.2.1 Sovereign NEXRAD Level-II (Archive II) decoder: bzip2 LDM blocks → Message-31 radials → REF/VEL/SW moments via (raw−offset)/scale, encoder for round-trip tests + AWS Open Data key builder.
ingest.observation 175.1.2.2–6 Radar Marshall-Palmer Z-R, bilinear grid remap, GOES ABI Planck brightness temperature, IMERG grid, METAR/SYNOP decoders, lightning flash gridding.
ingest.observation_clients 175.1.2.2–6 Real provider requests for public EUMETNET ORD OPERA composites; GOES-R, MTG and Himawari imagery; GPM IMERG; METAR/SYNOP/BUFR/ASOS; and GOES GLM/EUCLID lightning, with cadence/band validation and tiered archive keys.

Live-fetch seam#

Request construction, tiering, retry policy, and all format parsing/calibration run and are tested offline. The live network fetch is gated behind a credential/dependency seam (CredentialsMissing / DependencyMissing) — the optional ingest extra pulls cdsapi, xarray, dask, and kerchunk. No call fabricates a response.

Test#

bash
cd libs/gaia/weather-data && python3 -m pytest tests/ -q