Gaia Domain - Architecture

Planned architecture for the sovereign weather and climate ML platform.

Gaia is a planned data, model, and serving domain. It turns global weather and climate observations into operational forecast products and weather-impact APIs for the rest of Oshun. The domain does not exist in code yet — all packages, libraries, and services described here are designed in TODOS/phase-175.md and will be implemented in Phase 175.

Think of Gaia as three connected pipelines running end-to-end: (1) an ingestion pipeline that pulls raw observations from global sources and stores them in a weather data lake, (2) a model training pipeline that converts those observations into trained forecast models, and (3) a serving pipeline that runs those models on a regular 6-hour cadence and publishes forecast products through a REST API. Ten or more Oshun domains — from agriculture (Asase) to emergency response (Kuanyin) — read those products to make their own domain-specific decisions.

Ownership Boundary#

Gaia's responsibilities are defined by the data it ingests, the models it trains, the products it publishes, and the package namespace it controls.

Gaia owns:

• Earth-system data ingestion for reanalysis, operational NWP, radar, satellite, station, lightning, and cyclone archives.
• Weather and climate model training datasets, normalization statistics, splits, checkpoints, evaluation suites, and promotion gates.
• Forecast products, severe-weather alerts, downscaled products, and weather-impact APIs.
• The @gaia/* package namespace and planned libs/gaia/* libraries.

Several nearby domains could be confused with Gaia. The boundaries below make the split unambiguous:

Gaia does not own:

• Domain-specific decisions made from forecasts. Asase owns crop decisions, Oya owns mission planning, Cybele owns construction hardening, Lakshmi owns financial energy decisions, Uzume owns event operations, and Veritas owns editorial publication.
• Generic ML infrastructure. Nous owns reusable model serving, training, evaluation, GPU infrastructure, and model governance primitives. Gaia owns the weather-specific model architecture, data contracts, and skill gates.
• General astronomy. Nyx owns star catalogs, telescopes, ephemerides, and space exploration systems.

Planned Runtime Topology#

The diagram below shows the complete data and control flow, from external feed to downstream consumer. Each arrow is a service-to-service call or an event; each box is a planned @gaia/* package.

External data feeds
  ERA5 / ERA5-Land / HRES / GFS / MERRA-2 / IBTrACS
  NEXRAD / OPERA / GOES-R / MTG / Himawari / GPM / METAR / SYNOP / lightning
        |
        v
@gaia/weather-data
  GRIB2/Zarr conversion, Kerchunk indexes, temporal splits, statistics
        |
        v
Weather data lake
  MinIO object storage, Zarr chunks, STAC catalog, provenance hashes
        |
        +------------------+-------------------+------------------+
        |                  |                   |                  |
        v                  v                   v                  v
@gaia/graphcast     @gaia/gencast      @gaia/nowcast     @gaia/climate-emulator
deterministic       ensemble diffusion radar/satellite    long climate rollouts
forecasts           forecasts           nowcasts
        |                  |                   |                  |
        +------------------+-------------------+------------------+
                           |
                           v
@gaia/downscale and @gaia/cyclone-forecast
  site bias correction, fine-scale products, ATCF cyclone products
                           |
                           v
@gaia/forecast-serving
  REST/OpenAPI, signed output URLs, OGC/STAC products, CAP alerts, SLOs
                           |
                           v
Asase / Oya / Cybele / Galatea / Lakshmi / Uzume / Aphrodite / Veritas /
Demeter / Kuanyin / Oshun shell routines

Layer Responsibilities#

The system is organized into eight layers that track the flow of data from raw feed to consumer API. The table below maps each layer to its concrete responsibilities so engineers can quickly see which package owns which concern.

Data Architecture#

The data architecture is built around two principles: immutability of raw sources and efficient chunked access for training and serving. Together these guarantee that any model checkpoint can be traced back to the exact source data it was trained on.

• Raw source files are immutable and retained with source URL, provider, acquisition time, checksum, license, and access-policy metadata.
• Derived Zarr stores are chunked by variable, level, time, and grid so training and serving jobs can read contiguous forecast windows efficiently.
• WeatherSample is the canonical sample unit and must record valid time, lead time, grid specification, pressure levels, surface variables, normalization version, and provenance hash.
• Temporal splits must prevent leakage: historical train, validation, and test ranges are fixed by versioned dataset manifests.
• Forecast products are persisted as Zarr, GRIB-compatible export bundles, JSON summaries, STAC items, and OGC EDR/Features products where applicable.

Model Architecture#

Gaia plans six distinct model classes, each targeting a different forecasting regime. The architectures below were chosen to match the frontier AI-for-weather literature while giving Gaia sovereign, auditable weights.

• Deterministic forecasts use spherical GNNs over hierarchical icosahedral meshes with grid-to-mesh encoders, interaction-network processors, and mesh-to-grid decoders.
• Probabilistic forecasts use diffusion over residual world states conditioned on prior weather states, with calibrated ensemble samplers.
• Tropical-cyclone prediction adds differentiable heads for formation, track, intensity, pressure, wind-radius quadrants, and ATCF publication.
• Nowcasting combines radar, satellite, lightning, topography, and NWP boundary conditions for short-horizon precipitation.
• Climate emulation uses spherical FNO or ACE-class models for long rollouts under tunable forcing.
• Downscaling combines residual diffusion, station bias correction, quantile mapping, and urban heat-island modeling.

Operational Requirements#

The live operational cycle must meet hard latency, safety, and auditability requirements. These requirements exist because downstream domains rely on Gaia products for real-world safety decisions — a slow or incorrect forecast can affect drone no-fly zones, crop irrigation, live-event safety, and emergency alerts.

• Operational cycles run on 00/06/12/18 UTC cadence when live HRES or fallback GFS initial conditions are available.
• A cycle cannot publish if stability diagnostics, skill regression checks, or data-completeness gates fail.
• Severe-weather alerts must use a CAP 1.2-compatible profile, preserve source forecast identity, and emit audit events for downstream publication.
• p95 cycle completion target is under 20 minutes on the planned H100-class cluster for the operational deterministic cycle.
• Forecast availability, skill, latency, calibration, and downstream publication failures are first-class metrics.

Integration Architecture#

The boundary principle is strict: Gaia produces the forecast; every consumer owns the decision it makes from that forecast. This separation keeps Gaia's scope bounded and prevents logic duplication across domains — for example, both Asase and Oya need wind forecasts, but only Asase owns irrigation logic and only Oya owns drone no-fly logic.

Verification#

Gaia completion requires automated checks for schema validation, data ingestion integrity, train/validation/test leakage prevention, forecast API contracts, WeatherBench-2 and operational scorecards, tropical-cyclone verification, nowcast verification, red-team weather cases, SLO dashboards, and downstream adapter tests.

Source Coverage#

This architecture document was checked against TODOS/phase-175.md. It covers the planned topology, package layout, data/model/serving layers, cross-domain boundaries, operational cadence, downstream ownership, and verification gates defined there.