Eve library

phase-175.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 175: Sovereign Weather & Climate ML — Spherical GNN Forecasting, Probabilistic Diffusion Climate Models, Direct Cyclone Prediction, Nowcasting, Climate Emulation & Downscaling#

End-to-end sovereign replication and extension of the frontier AI-for-Earth-sciences stack pioneered by Google DeepMind (GraphCast, GenCast, FGN, WeatherNext), Huawei (Pangu-Weather), NVIDIA (FourCastNet, StormCast), and Microsoft/Aurora. Targets medium-range deterministic forecasting, ensemble probabilistic forecasting, tropical-cyclone direct prediction, minute-scale precipitation nowcasting, decadal climate emulation, and statistical downscaling — all from a single unified data, training, and serving pipeline. Integrates with @asase/agriculture, @oya/drones, @cybele/construction, @galatea/fleet, and @veritas/news via sovereign weather APIs.

Tech split: Python (~60%) for model training (PyTorch + JAX/Haiku parity) and reanalysis ingestion; Rust (~20%) for Zarr/GRIB conversion and operational ensemble workers; TypeScript (~20%) for orchestration, forecast APIs, and impact integration.

Library layout: libs/gaia/weather-data/, libs/gaia/graphcast/, libs/gaia/gencast/, libs/gaia/cyclone-forecast/, libs/gaia/nowcast/, libs/gaia/climate-emulator/, libs/gaia/downscale/, libs/gaia/forecast-serving/.


175.1 Reanalysis & Observation Data Foundation#

175.1.1 Reanalysis Archive Ingestion#

  • 175.1.1.1 Build @gaia/weather-data ERA5 ingestion client against Copernicus CDS (cdsapi), pulling 1940–present hourly 0.25° surface fields (2m temperature, 10m u/v winds, MSLP, total precipitation, 2m dewpoint) and 37 pressure levels (geopotential, u/v, w, temperature, specific humidity) with GRIB2→Zarr conversion via xarray + dask + kerchunk, tiered MinIO storage (hot: last 5y, warm: 1990–, cold: 1940–1989)
  • 175.1.1.2 Build ERA5-Land high-resolution surface ingestion (0.1° hourly, 1950–present) for downscaling priors and hydrology features
  • 175.1.1.3 Build IFS HRES operational analysis ingestion (ECMWF MARS, 9 km deterministic, 0/6/12/18 UTC cycles) for live initial conditions
  • 175.1.1.4 Build GFS operational ingestion (NOAA NOMADS, 13 km, four cycles) as fallback initial-condition source and cross-validation
  • 175.1.1.5 Build MERRA-2 ingestion (NASA GES DISC, 1980–present, 0.5° × 0.625°) for multi-reanalysis agreement scoring and aerosol inputs
  • 175.1.1.6 Build IBTrACS tropical-cyclone best-track ingestion (1842–present, all basins) with 6-hourly position, wind, pressure, radius fields — ground truth for cyclone-model training and evaluation

175.1.2 Observation & Radar Ingestion#

  • 175.1.2.1 Build NEXRAD Level-II ingestion (US radar mosaic, 250 m, 2-min cadence) via AWS Open Data — primary nowcasting dataset
  • 175.1.2.2 Build OPERA composite ingestion (European radar, 2 km, 5-min) for European nowcasting coverage
  • 175.1.2.3 Build GOES-R / MTG / Himawari geostationary satellite ingestion (16 bands, 10-min full disk) with CSPP / SatPy radiometric calibration and remapping to the model grid
  • 175.1.2.4 Build GPM IMERG satellite precipitation ingestion (0.1° half-hourly) for global nowcasting and tropical regions without radar
  • 175.1.2.5 Build automatic station network ingestion (METAR, SYNOP, BUFR, and ASOS) for verification, assimilation, and station-level bias correction
  • 175.1.2.6 Build lightning detection ingestion (GLM from GOES, EUCLID for Europe) for convective nowcasting labels and severe-weather triggers

175.1.3 Data Catalogue & Sampling#

  • 175.1.3.1 Define canonical WeatherSample schema — date, lead-time, grid specification (lat/lon or icosahedral level), pressure levels, surface variables, and provenance hash — with Zod validation and Zarr chunk alignment
  • 175.1.3.2 Build stratified training-sample loader (balanced across seasons, basins, extreme-event classes from IBTrACS / SPI / heat-wave indices) and curriculum scheduler that up-weights tropical cyclones, atmospheric rivers, and blocking events
  • 175.1.3.3 Build rolling train/val/test split enforcing strict temporal separation (train ≤2018, val 2019–2020, test 2021+) to prevent leakage through persistence and climate drift
  • 175.1.3.4 Build statistics package (per-variable, per-level mean / std / min / max) for model normalisation and denormalisation, stored versioned alongside each trained checkpoint

175.2 GraphCast-Class Deterministic Spherical GNN Forecasting#

175.2.1 Mesh & Encoder/Processor/Decoder#

  • 175.2.1.1 Build icosahedral multi-mesh generator (M0 icosahedron → M6 refinement with 40 962 nodes) with uniform-area triangular faces and pre-computed edge lists for every refinement level
  • 175.2.1.2 Implement Grid→Mesh encoder: for each mesh node, attend to nearby 0.25° grid nodes (radius ≈ 0.6×mesh edge length) via an MLP on (node_feature, grid_feature, relative_position) — identical semantics to the public GraphCast encoder
  • 175.2.1.3 Implement Mesh processor: 16 rounds of interaction-network message passing over all 6 refinement levels simultaneously (hierarchical multi-mesh) with residual updates and layer norm — produce updated mesh node embeddings
  • 175.2.1.4 Implement Mesh→Grid decoder: for each grid node, attend to its containing mesh triangle's 3 nodes via MLP, predict residual tendency relative to input state; add to input to produce +6h forecast
  • 175.2.1.5 Implement autoregressive rollout driver (6h step × up to 40 steps = 10-day horizon) with checkpoint-supported gradient accumulation for multi-step fine-tuning

175.2.2 Training & Scaling#

  • 175.2.2.1 Train GraphCast-Sovereign base (37M params, 1° ERA5): single-step MSE for 300k steps, then multi-step rollout fine-tune. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.2.2.2 Train GraphCast-Sovereign-XL (0.25°, 226M params) on a TPU-v5e / H100 pod with FSDP2 and BF16. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.2.2.3 Build ensemble-of-Sovereigns (10 random seeds × different training subsets) for cheap deterministic ensemble baseline
  • 175.2.2.4 Implement per-variable, per-level, per-latitude loss weighting (area-weighted cos(lat) + published GraphCast variable weights) — critical for good polar and upper-air skill
  • 175.2.2.5 Build rollout-stability diagnostics (energy conservation, spectrum at T+240h vs ERA5 climatology, blur detector) and automatic early-stop on spectral collapse

175.3 GenCast-Class Probabilistic Diffusion Forecasting#

175.3.1 Diffusion World-State Model#

  • 175.3.1.1 Implement state-diffusion model operating on +12h residuals over the icosahedral mesh (EDM parametrisation: preconditioned score-matching, ρ=7, σ_min 0.02, σ_max 88) with the Phase 175.2 GNN as the score network backbone conditioned on the previous two states
  • 175.3.1.2 Implement deterministic DPM-Solver++ sampler for 20-step ensemble generation (target parity with published GenCast 20-step skill) and stochastic Heun sampler for calibrated tail sampling
  • 175.3.1.3 Train the ensemble-diffusion model on 1979–2018 ERA5 12h residuals with an EDM-weighted loss and validate CRPS vs IFS-ENS. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.3.1.4 Implement 50-member operational ensemble generation on single H100 / B200 (target ≤ 8 minutes wall-clock for 15-day 50-member forecast) with batched multi-seed sampling

175.3.2 Calibration & Diagnostics#

  • 175.3.2.1 Build CRPS, spread-skill ratio, rank histogram, and reliability-diagram calibration suite — computed daily on latest forecasts vs ERA5 analysis
  • 175.3.2.2 Build extreme-percentile verification (P99.9 precipitation, heat-wave return periods, wind-speed tails) — guards against distributional shortcut learning
  • 175.3.2.3 Build ensemble-member clustering and scenario extraction (k-means on principal components of +240h states) for narrative forecast-product generation

175.4 FGN-Class Direct Tropical Cyclone Prediction#

175.4.1 Functional Generative Cyclone Network#

  • 175.4.1.1 Define cyclone functional output — at each 6h lead time, predict (probability-of-cyclone, center lat/lon, max sustained wind, central MSLP, RMW, R34/R50/R64 quadrants, eye formation flag) jointly as a differentiable head conditioned on the diffusion backbone latent
  • 175.4.1.2 Implement trajectory head: Transformer decoder over predicted cyclone centres conditioned on past 24h IBTrACS track + model latent, outputting 0–120h trajectory distribution with stochastic sampling
  • 175.4.1.3 Train the cyclone model end-to-end with the combined position/intensity/category loss. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.4.1.4 Build automated NHC-compatible ATCF output generator (tech format: BASIN, CY, YYYYMMDDHH, TAU, LatN/S, LonE/W, Vmax, MSLP, category, R34/R50/R64) for ingestion by downstream emergency-response systems
  • 175.4.1.5 Integrate with @oya/drones (hurricane hunter drone swarm tasking), @cybele/construction (pre-storm asset hardening alerts), and @veritas/news (automated tropical-cyclone bulletin desk)

175.5 Minute-Scale Precipitation Nowcasting#

175.5.1 Radar Nowcasting Model#

  • 175.5.1.1 Implement MetNet-3-class axial transformer over stacked radar composites (last 90 min, 2-min cadence) + geostationary satellite + topography + NWP boundary conditions → 0–12 h precipitation at 1 km, 5-min resolution
  • 175.5.1.2 Implement DGMR-class conditional GAN for ensemble precipitation nowcasting (4-member radar ensemble, skill against pySTEPS optical flow and operational UK Met Office nowcast)
  • 175.5.1.3 Build convective-mode classifier (stratiform vs convective vs tropical vs frontal) from radar + satellite + lightning for downstream model-mixture weighting
  • 175.5.1.4 Build CSI, POD, FAR, and FSS verification harness at 0.1/1/5/10 mm h⁻¹ thresholds versus StageIV / radar-gauge blends

175.6 Climate Emulation & Long Rollouts#

175.6.1 ACE-Class Climate Emulator#

  • 175.6.1.1 Train the spherical-FNO / ACE-class climate emulator on CMIP6 control + SSP runs for 100-year rollouts. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.6.1.2 Build coupled sea-ice-albedo / soil-moisture feedback modules via FNO-in-FNO composition to preserve multi-scale feedbacks
  • 175.6.1.3 Implement decadal hindcast driver (1960–2020) validated against HadCRUT5, GISTEMP, AMIP-SST
  • 175.6.1.4 Build scenario emulator with tunable forcing (CO₂, CH₄, aerosol) for rapid what-if decision support, exposed via forecast API

175.7 Fine-Scale Downscaling & Bias Correction#

175.7.1 Statistical / ML Downscaling#

  • 175.7.1.1 Train the CorrDiff-class residual-diffusion downscaling model (25 km → 2 km) conditioned on the coarse forecasts. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). blocked:governance
  • 175.7.1.2 Train station-bias correction MLP/XGBoost heads per METAR/SYNOP station consuming the 25 km ensemble for site-specific forecast calibration (1–10 day)
  • 175.7.1.3 Build multivariate quantile-mapping bias correction for derived variables (wind-power density, heating/cooling-degree days, growing-degree days) used by @asase/agriculture and energy agents
  • 175.7.1.4 Build urban-downscaling module (50 m, diurnal UHI-aware) on top of European Urban Atlas + OSM land-use for @cybele/construction and @themis/governance climate-adaptation products

175.8 Operational Forecast Serving & Integration#

175.8.1 Forecast APIs & SLAs#

  • 175.8.1.1 Build @gaia/forecast-serving BFF exposing /forecast/deterministic, /forecast/ensemble, /forecast/cyclone, /forecast/nowcast, /forecast/climate-scenario with OpenAPI 3.1 specs, per-tenant rate limiting, and signed-URL GRIB/Zarr output streaming
  • 175.8.1.2 Build live operational cycle runner (four daily cycles aligned with 00/06/12/18 UTC HRES analysis) with automatic initial-condition fetch, forecast launch, publication, and rollback on stability-diagnostic failure
  • 175.8.1.3 Build forecast-product warehouse (Zarr on MinIO with STAC 1.0 catalogue, OGC EDR / OGC API Features endpoints) for external scientific users and @veritas/news editorial system
  • 175.8.1.4 Build alerting integration with CAP 1.2 profile publication (urn:oid:2.49.0.1.840.1) for severe-weather events — wired to @kuanyin/emergency and @oshun/shell-routines weather alerts
  • 175.8.1.5 Build latency / skill / availability SLO dashboard (Grafana) — p95 cycle completion < 20 min on H100-40 cluster, 24h deterministic RMSE within 2% of published GraphCast baseline

175.8.2 Downstream Domain Integration#

  • 175.8.2.1 Wire forecasts into @asase/agriculture crop-model ensemble and irrigation/planting decision agent — quantify yield-lift from ML weather vs IFS HRES baseline in A/B
  • 175.8.2.2 Wire cyclone-forecast into @oya/drones mission planner (no-fly cones, evacuation-swarm tasking) and @galatea/fleet outdoor robot safety gating
  • 175.8.2.3 Wire renewable-energy potential (solar GHI, wind 100m, hydro-inflow) into @lakshmi/finance energy trading / capacity planning
  • 175.8.2.4 Wire nowcast + lightning into @uzume/live-performance outdoor-event weather dashboard and @aphrodite/streaming outdoor shoot risk scoring

175.9 Data Flywheel & Evaluation#

175.9.1 Continuous Improvement#

  • 175.9.1.1 Extend Phase 85 data flywheel with a weather-outcomes collector — forecast vs analysis deltas per cycle, per variable, per lead-time, per region — continuously fed back into training-data difficulty weighting (Phase 89.4.1.8) and retraining triggers (Phase 85.17.1.5)
  • 175.9.1.2 Build public-benchmark suite (WeatherBench-2 + ECMWF scorecards + NHC-verification + pysteps-verification) with automated leaderboard publication per checkpoint to the Phase 91 evaluation hub
  • 175.9.1.3 Build red-team / stress-test suite: hurricane Ian 2022, Hurricane Lee 2023, 2021 Pacific Northwest heat dome, February 2021 Texas cold snap, July 2022 UK record heat — require every new checkpoint to match or exceed the prior on all cases before promotion
  • 175.9.1.4 Build observational-obs feedback — compare operational forecasts against surface station / radiosonde / buoy arrivals in real time, feed per-station residuals back into 175.7.1.2 station-bias heads
  • 175.9.1.5 Publish a scientific paper and open-weights release for each major model class. Needs an owner decision on a compute budget — this is pod-scale weather-model training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available but the training compute is not (stated 2026-09-18). It also depends on the trained models existing and is a human authoring/release act. blocked:governance