Historical task source. The Eve task board holds current task state.
Phase 175: Sovereign Weather & Climate ML — Spherical GNN Forecasting, Probabilistic Diffusion Climate Models, Direct Cyclone Prediction, Nowcasting, Climate Emulation & Downscaling#
End-to-end sovereign replication and extension of the frontier
AI-for-Earth-sciences stack pioneered by Google DeepMind (GraphCast, GenCast,
FGN, WeatherNext), Huawei (Pangu-Weather), NVIDIA (FourCastNet, StormCast), and
Microsoft/Aurora. Targets medium-range deterministic forecasting, ensemble
probabilistic forecasting, tropical-cyclone direct prediction, minute-scale
precipitation nowcasting, decadal climate emulation, and statistical downscaling
— all from a single unified data, training, and serving pipeline. Integrates
with @asase/agriculture, @oya/drones, @cybele/construction,
@galatea/fleet, and @veritas/news via sovereign weather APIs.
Tech split: Python (~60%) for model training (PyTorch + JAX/Haiku parity) and reanalysis ingestion; Rust (~20%) for Zarr/GRIB conversion and operational ensemble workers; TypeScript (~20%) for orchestration, forecast APIs, and impact integration.
Library layout: libs/gaia/weather-data/, libs/gaia/graphcast/,
libs/gaia/gencast/, libs/gaia/cyclone-forecast/, libs/gaia/nowcast/,
libs/gaia/climate-emulator/, libs/gaia/downscale/,
libs/gaia/forecast-serving/.
175.1 Reanalysis & Observation Data Foundation#
175.1.1 Reanalysis Archive Ingestion#
- 175.1.1.1 Build
@gaia/weather-dataERA5 ingestion client against Copernicus CDS (cdsapi), pulling 1940–present hourly 0.25° surface fields (2m temperature, 10m u/v winds, MSLP, total precipitation, 2m dewpoint) and 37 pressure levels (geopotential, u/v, w, temperature, specific humidity) with GRIB2→Zarr conversion viaxarray + dask + kerchunk, tiered MinIO storage (hot: last 5y, warm: 1990–, cold: 1940–1989) - 175.1.1.2 Build ERA5-Land high-resolution surface ingestion (0.1° hourly, 1950–present) for downscaling priors and hydrology features
- 175.1.1.3 Build IFS HRES operational analysis ingestion (ECMWF MARS, 9 km deterministic, 0/6/12/18 UTC cycles) for live initial conditions
- 175.1.1.4 Build GFS operational ingestion (NOAA NOMADS, 13 km, four cycles) as fallback initial-condition source and cross-validation
- 175.1.1.5 Build MERRA-2 ingestion (NASA GES DISC, 1980–present, 0.5° × 0.625°) for multi-reanalysis agreement scoring and aerosol inputs
- 175.1.1.6 Build IBTrACS tropical-cyclone best-track ingestion (1842–present, all basins) with 6-hourly position, wind, pressure, radius fields — ground truth for cyclone-model training and evaluation
175.1.2 Observation & Radar Ingestion#
- 175.1.2.1 Build NEXRAD Level-II ingestion (US radar mosaic, 250 m, 2-min cadence) via AWS Open Data — primary nowcasting dataset
- 175.1.2.2 Build OPERA composite ingestion (European radar, 2 km, 5-min) for European nowcasting coverage
- 175.1.2.3 Build GOES-R / MTG / Himawari geostationary satellite ingestion (16 bands, 10-min full disk) with CSPP / SatPy radiometric calibration and remapping to the model grid
- 175.1.2.4 Build GPM IMERG satellite precipitation ingestion (0.1° half-hourly) for global nowcasting and tropical regions without radar
- 175.1.2.5 Build automatic station network ingestion (METAR, SYNOP, BUFR, and ASOS) for verification, assimilation, and station-level bias correction
- 175.1.2.6 Build lightning detection ingestion (GLM from GOES, EUCLID for Europe) for convective nowcasting labels and severe-weather triggers
175.1.3 Data Catalogue & Sampling#
- 175.1.3.1 Define canonical
WeatherSampleschema — date, lead-time, grid specification (lat/lon or icosahedral level), pressure levels, surface variables, and provenance hash — with Zod validation and Zarr chunk alignment - 175.1.3.2 Build stratified training-sample loader (balanced across seasons, basins, extreme-event classes from IBTrACS / SPI / heat-wave indices) and curriculum scheduler that up-weights tropical cyclones, atmospheric rivers, and blocking events
- 175.1.3.3 Build rolling train/val/test split enforcing strict temporal separation (train ≤2018, val 2019–2020, test 2021+) to prevent leakage through persistence and climate drift
- 175.1.3.4 Build statistics package (per-variable, per-level mean / std / min / max) for model normalisation and denormalisation, stored versioned alongside each trained checkpoint
175.2 GraphCast-Class Deterministic Spherical GNN Forecasting#
175.2.1 Mesh & Encoder/Processor/Decoder#
- 175.2.1.1 Build icosahedral multi-mesh generator (M0 icosahedron → M6 refinement with 40 962 nodes) with uniform-area triangular faces and pre-computed edge lists for every refinement level
- 175.2.1.2 Implement Grid→Mesh encoder: for each mesh node, attend to nearby 0.25° grid nodes (radius ≈ 0.6×mesh edge length) via an MLP on (node_feature, grid_feature, relative_position) — identical semantics to the public GraphCast encoder
- 175.2.1.3 Implement Mesh processor: 16 rounds of interaction-network message passing over all 6 refinement levels simultaneously (hierarchical multi-mesh) with residual updates and layer norm — produce updated mesh node embeddings
- 175.2.1.4 Implement Mesh→Grid decoder: for each grid node, attend to its containing mesh triangle's 3 nodes via MLP, predict residual tendency relative to input state; add to input to produce +6h forecast
- 175.2.1.5 Implement autoregressive rollout driver (6h step × up to 40 steps = 10-day horizon) with checkpoint-supported gradient accumulation for multi-step fine-tuning
175.2.2 Training & Scaling#
- 175.2.2.1 Train GraphCast-Sovereign base (37M params, 1° ERA5):
single-step MSE for 300k steps, then multi-step rollout fine-tune. Needs
an owner decision on a compute budget — this is pod-scale weather-model
training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs
are openly available but the training compute is not (stated 2026-09-18).
blocked:governance - 175.2.2.2 Train GraphCast-Sovereign-XL (0.25°, 226M params) on a
TPU-v5e / H100 pod with FSDP2 and BF16. Needs an owner decision on a
compute budget — this is pod-scale weather-model training (TPU-v5e /
H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available
but the training compute is not (stated 2026-09-18).
blocked:governance - 175.2.2.3 Build ensemble-of-Sovereigns (10 random seeds × different training subsets) for cheap deterministic ensemble baseline
- 175.2.2.4 Implement per-variable, per-level, per-latitude loss weighting (area-weighted cos(lat) + published GraphCast variable weights) — critical for good polar and upper-air skill
- 175.2.2.5 Build rollout-stability diagnostics (energy conservation, spectrum at T+240h vs ERA5 climatology, blur detector) and automatic early-stop on spectral collapse
175.3 GenCast-Class Probabilistic Diffusion Forecasting#
175.3.1 Diffusion World-State Model#
- 175.3.1.1 Implement state-diffusion model operating on +12h residuals over the icosahedral mesh (EDM parametrisation: preconditioned score-matching, ρ=7, σ_min 0.02, σ_max 88) with the Phase 175.2 GNN as the score network backbone conditioned on the previous two states
- 175.3.1.2 Implement deterministic DPM-Solver++ sampler for 20-step ensemble generation (target parity with published GenCast 20-step skill) and stochastic Heun sampler for calibrated tail sampling
- 175.3.1.3 Train the ensemble-diffusion model on 1979–2018 ERA5 12h
residuals with an EDM-weighted loss and validate CRPS vs IFS-ENS. Needs an
owner decision on a compute budget — this is pod-scale weather-model
training (TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs
are openly available but the training compute is not (stated 2026-09-18).
blocked:governance - 175.3.1.4 Implement 50-member operational ensemble generation on single H100 / B200 (target ≤ 8 minutes wall-clock for 15-day 50-member forecast) with batched multi-seed sampling
175.3.2 Calibration & Diagnostics#
- 175.3.2.1 Build CRPS, spread-skill ratio, rank histogram, and reliability-diagram calibration suite — computed daily on latest forecasts vs ERA5 analysis
- 175.3.2.2 Build extreme-percentile verification (P99.9 precipitation, heat-wave return periods, wind-speed tails) — guards against distributional shortcut learning
- 175.3.2.3 Build ensemble-member clustering and scenario extraction (k-means on principal components of +240h states) for narrative forecast-product generation
175.4 FGN-Class Direct Tropical Cyclone Prediction#
175.4.1 Functional Generative Cyclone Network#
- 175.4.1.1 Define cyclone functional output — at each 6h lead time, predict (probability-of-cyclone, center lat/lon, max sustained wind, central MSLP, RMW, R34/R50/R64 quadrants, eye formation flag) jointly as a differentiable head conditioned on the diffusion backbone latent
- 175.4.1.2 Implement trajectory head: Transformer decoder over predicted cyclone centres conditioned on past 24h IBTrACS track + model latent, outputting 0–120h trajectory distribution with stochastic sampling
- 175.4.1.3 Train the cyclone model end-to-end with the combined
position/intensity/category loss. Needs an owner decision on a compute
budget — this is pod-scale weather-model training (TPU-v5e / H100-class),
not a coding task; the ERA5/CMIP6 inputs are openly available but the
training compute is not (stated 2026-09-18).
blocked:governance - 175.4.1.4 Build automated NHC-compatible ATCF output generator (tech format: BASIN, CY, YYYYMMDDHH, TAU, LatN/S, LonE/W, Vmax, MSLP, category, R34/R50/R64) for ingestion by downstream emergency-response systems
- 175.4.1.5 Integrate with
@oya/drones(hurricane hunter drone swarm tasking),@cybele/construction(pre-storm asset hardening alerts), and@veritas/news(automated tropical-cyclone bulletin desk)
175.5 Minute-Scale Precipitation Nowcasting#
175.5.1 Radar Nowcasting Model#
- 175.5.1.1 Implement MetNet-3-class axial transformer over stacked radar composites (last 90 min, 2-min cadence) + geostationary satellite + topography + NWP boundary conditions → 0–12 h precipitation at 1 km, 5-min resolution
- 175.5.1.2 Implement DGMR-class conditional GAN for ensemble precipitation nowcasting (4-member radar ensemble, skill against pySTEPS optical flow and operational UK Met Office nowcast)
- 175.5.1.3 Build convective-mode classifier (stratiform vs convective vs tropical vs frontal) from radar + satellite + lightning for downstream model-mixture weighting
- 175.5.1.4 Build CSI, POD, FAR, and FSS verification harness at 0.1/1/5/10 mm h⁻¹ thresholds versus StageIV / radar-gauge blends
175.6 Climate Emulation & Long Rollouts#
175.6.1 ACE-Class Climate Emulator#
- 175.6.1.1 Train the spherical-FNO / ACE-class climate emulator on
CMIP6 control + SSP runs for 100-year rollouts. Needs an owner decision on
a compute budget — this is pod-scale weather-model training (TPU-v5e /
H100-class), not a coding task; the ERA5/CMIP6 inputs are openly available
but the training compute is not (stated 2026-09-18).
blocked:governance - 175.6.1.2 Build coupled sea-ice-albedo / soil-moisture feedback modules via FNO-in-FNO composition to preserve multi-scale feedbacks
- 175.6.1.3 Implement decadal hindcast driver (1960–2020) validated against HadCRUT5, GISTEMP, AMIP-SST
- 175.6.1.4 Build scenario emulator with tunable forcing (CO₂, CH₄, aerosol) for rapid what-if decision support, exposed via forecast API
175.7 Fine-Scale Downscaling & Bias Correction#
175.7.1 Statistical / ML Downscaling#
- 175.7.1.1 Train the CorrDiff-class residual-diffusion downscaling
model (25 km → 2 km) conditioned on the coarse forecasts. Needs an owner
decision on a compute budget — this is pod-scale weather-model training
(TPU-v5e / H100-class), not a coding task; the ERA5/CMIP6 inputs are
openly available but the training compute is not (stated 2026-09-18).
blocked:governance - 175.7.1.2 Train station-bias correction MLP/XGBoost heads per METAR/SYNOP station consuming the 25 km ensemble for site-specific forecast calibration (1–10 day)
- 175.7.1.3 Build multivariate quantile-mapping bias correction for
derived variables (wind-power density, heating/cooling-degree days,
growing-degree days) used by
@asase/agricultureand energy agents - 175.7.1.4 Build urban-downscaling module (50 m, diurnal UHI-aware) on
top of European Urban Atlas + OSM land-use for
@cybele/constructionand@themis/governanceclimate-adaptation products
175.8 Operational Forecast Serving & Integration#
175.8.1 Forecast APIs & SLAs#
- 175.8.1.1 Build
@gaia/forecast-servingBFF exposing/forecast/deterministic,/forecast/ensemble,/forecast/cyclone,/forecast/nowcast,/forecast/climate-scenariowith OpenAPI 3.1 specs, per-tenant rate limiting, and signed-URL GRIB/Zarr output streaming - 175.8.1.2 Build live operational cycle runner (four daily cycles aligned with 00/06/12/18 UTC HRES analysis) with automatic initial-condition fetch, forecast launch, publication, and rollback on stability-diagnostic failure
- 175.8.1.3 Build forecast-product warehouse (Zarr on MinIO with STAC
1.0 catalogue, OGC EDR / OGC API Features endpoints) for external
scientific users and
@veritas/newseditorial system - 175.8.1.4 Build alerting integration with CAP 1.2 profile publication
(
urn:oid:2.49.0.1.840.1) for severe-weather events — wired to@kuanyin/emergencyand@oshun/shell-routinesweather alerts - 175.8.1.5 Build latency / skill / availability SLO dashboard (Grafana) — p95 cycle completion < 20 min on H100-40 cluster, 24h deterministic RMSE within 2% of published GraphCast baseline
175.8.2 Downstream Domain Integration#
- 175.8.2.1 Wire forecasts into
@asase/agriculturecrop-model ensemble and irrigation/planting decision agent — quantify yield-lift from ML weather vs IFS HRES baseline in A/B - 175.8.2.2 Wire cyclone-forecast into
@oya/dronesmission planner (no-fly cones, evacuation-swarm tasking) and@galatea/fleetoutdoor robot safety gating - 175.8.2.3 Wire renewable-energy potential (solar GHI, wind 100m,
hydro-inflow) into
@lakshmi/financeenergy trading / capacity planning - 175.8.2.4 Wire nowcast + lightning into
@uzume/live-performanceoutdoor-event weather dashboard and@aphrodite/streamingoutdoor shoot risk scoring
175.9 Data Flywheel & Evaluation#
175.9.1 Continuous Improvement#
- 175.9.1.1 Extend Phase 85 data flywheel with a
weather-outcomescollector — forecast vs analysis deltas per cycle, per variable, per lead-time, per region — continuously fed back into training-data difficulty weighting (Phase 89.4.1.8) and retraining triggers (Phase 85.17.1.5) - 175.9.1.2 Build public-benchmark suite (WeatherBench-2 + ECMWF scorecards + NHC-verification + pysteps-verification) with automated leaderboard publication per checkpoint to the Phase 91 evaluation hub
- 175.9.1.3 Build red-team / stress-test suite: hurricane Ian 2022, Hurricane Lee 2023, 2021 Pacific Northwest heat dome, February 2021 Texas cold snap, July 2022 UK record heat — require every new checkpoint to match or exceed the prior on all cases before promotion
- 175.9.1.4 Build observational-obs feedback — compare operational forecasts against surface station / radiosonde / buoy arrivals in real time, feed per-station residuals back into 175.7.1.2 station-bias heads
- 175.9.1.5 Publish a scientific paper and open-weights release for each
major model class. Needs an owner decision on a compute budget — this is
pod-scale weather-model training (TPU-v5e / H100-class), not a coding
task; the ERA5/CMIP6 inputs are openly available but the training compute
is not (stated 2026-09-18). It also depends on the trained models existing
and is a human authoring/release act.
blocked:governance