# V1 SLOs and Disaster Recovery Objectives Status: planning baseline, created 2026-06-12 to close `V1_V7_PLAN_SET_AUDIT_2026-06-12.md` §6.2 (V1: "no availability SLOs (latency budgets exist, uptime budgets don't)") and §6.1(6) (no DR/backup objectives). Latency budgets cited below are the spec's own; the availability targets and RPO/RTO figures are planning assumptions adopted 2026-06-12, each with its derivation, to be ratified at the GA go/no-go. ## 1. Availability SLOs per surface (monthly windows) Tiering principle: surfaces that gate **money, identity, safety, or institutional contracts** get 99.9; realtime/generative surfaces get 99.5 with honest degraded modes; the GPU render plane gets 99.0 because the spec already defines a degradation ladder that preserves the session (`V1/features.md:4420-4422`) — availability money is better spent on the ladder than on gold-plating the render plane. | Surface | SLI definition | Target | Monthly error budget | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -------------------- | | **BFF API** (`apps/oshun/bff`) | non-5xx responses ÷ valid requests; latency SLI: p99 ≤ 800 ms reads / 1.5 s writes | **99.9 %** | 43 min 49 s | | **Auth/session issuance** | successful token issuance ÷ valid attempts | **99.9 %** | 43 min 49 s | | **Psyche realtime — platform** | session-minutes in which a session could be _established or continued in any mode_ (text counts — the fallback chain is the design, `V1/features.md:2239-2246`) | **99.5 %** | 3 h 39 min | | **Psyche realtime — voice mode** | session-minutes with voice available within budget (degradation to text counts against this SLI but not the platform SLI) | **99.0 %** | 7 h 18 min | | **Living Scenes control plane** (Conductor session establishment, score CRUD, cue gate) | successful session starts + cue acks ÷ attempts | **99.5 %** | 3 h 39 min | | **Living Scenes render plane** | session-minutes not in BLACK queue state (CAPACITY_MODEL §6.2); first-frame budget compliance ≥ 95 % of sessions | **99.0 %** | 7 h 18 min | | **Metis gateway** (`apps/metis/api-gateway`; LTI launches, gradebook emission) | non-5xx ÷ valid; LTI launch success ratio | **99.9 %** (institutional contracts quote this number) | 43 min 49 s | | **Public viewer / attestation pages** (`oshun.app/scene/`) | successful page render ÷ requests (CDN + static-first design) | **99.95 %** | 21 min 54 s | | **Payments invoice plane** (invoice issuance + payment-detection pipeline) | invoice issuance success; per-chain detection freshness: chain tip lag ≤ 2 blocks for 99 % of minutes (RISK_REGISTER R-13 monitor) | **99.9 %** | 43 min 49 s | | **Event bus delivery** (`@oshun/event-bus` — Redis pub/sub with TTL-keyed replay) | events delivered-at-least-once within 60 s ÷ published | **99.9 %** | 43 min 49 s | Notes: - Gradebook emission is asynchronous with idempotent re-emission (`V1/features.md:1569-1572`) — its SLO is _freshness_ (99 % of entries emitted ≤ 15 min after trigger), carried under the Metis gateway row. - Crisis-path availability is **not** an SLO with a budget: the crisis frame and cue kill path fail closed and have reserved capacity (CAPACITY_MODEL §6.2); any crisis-path unavailability is a Sev-1 regardless of duration. ## 2. Error-budget policy - Burn-rate alerts per the multiwindow standard: page at 14.4× budget burn over 1 h, ticket at 6× over 6 h (both windows must agree). - Budget exhausted → feature freeze on that surface: only reliability/rollback changes merge until the 30-day window recovers; canary promotions (LAUNCH_TIMELINE Phase 3) are blocked while any RED-tier surface is over budget. - SLO review monthly; targets ratchet only after two consecutive months of ≥ 50 % unspent budget. ## 3. DR objectives per data class Region posture context (§5) — per-data-plane primary/standby. "RPO" is data loss tolerance; "RTO" time-to-restore service on that class. | Data class | Store | Failure-domain RPO/RTO (AZ or instance loss) | Region-loss RPO/RTO | Rationale | | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Per-domain OLTP (tara, nisaba, arete, nyx, veritas, metis, lilith, isis, sophia, hathor, bellona; `V1/ARCHITECTURE.md:1195-1202`) | RDS Postgres multi-AZ, PITR | **RPO ≤ 5 min / RTO ≤ 60 min** (multi-AZ failover itself ≤ 5 min) | **RPO ≤ 15 min / RTO ≤ 4 h** (cross-region replica for Tier-1 DBs: identity/entitlements, audit, metis; snapshot-restore for the rest, RTO ≤ 12 h) | 5-min WAL shipping is RDS-native; Tier-1 split keeps cross-region replica cost on the records that are contractually or financially irreplaceable | | Audit platform (`@oshun/audit-platform`, append-only) | Postgres + hourly export | **RPO ≤ 1 min** (synchronous standby) / RTO ≤ 60 min | RPO ≤ 60 min via hourly batch export to S3 **Object Lock (WORM)** | audit is the substrate for DSAR, FERPA access, payment disputes — near-zero loss tolerance, and WORM defends against the erase-your-tracks attacker | | Kept-artifact **Scores + Render Envelopes + Cue Plans** (source of truth, kilobytes each, `V1/features.md:4195-4199`) | Postgres (yemaya) + nightly export | RPO ≤ 5 min / RTO ≤ 60 min | RPO ≤ 15 min / RTO ≤ 4 h (Tier-1) | "offerings persist for the lifetime of the user account" (`V1/features.md:4206-4208`) makes these irreplaceable; the MP4s are not | | Materialized MP4 caches + share renders | S3 (MinIO in dev) | RPO 24 h / RTO = re-render on demand | same | re-renderable bit-equivalently from Score + Envelope by the determinism gate (`V1/features.md:4193-4199`) — DR by recomputation is the _designed_ path | | Source media, ingestion originals, model artifacts | S3, versioned | RPO ≤ 15 min (S3 versioning + same-region durability) / RTO ≤ 2 h | cross-region replication on `sources` and `provenance` buckets, RPO ≤ 15 min | originals are not recomputable | | Redis — cache tier | ElastiCache | RPO = n/a (rebuild) / RTO ≤ 15 min warm-up | rebuild | cache by definition | | Redis — **event bus** (`@oshun/event-bus`) | ElastiCache, AOF everysec | **RPO ≤ 1 s–1 min / RTO ≤ 30 min** | not replicated cross-region — gap re-driven from the **transactional outbox** | consumers are idempotent with dedup/replay required by the test matrix (`V1/features.md:6169-6170`); critical producers (payments-bridge entitlement grants, gradebook emission) write an outbox row in their domain Postgres in the same transaction, so the bus is never the only copy of a money- or grade-bearing event | | Search/vector/graph indexes (Elasticsearch, Qdrant, Neo4j; `V1/ARCHITECTURE.md:1213-1219`) | managed/EC2 | RPO n/a (derived data) / RTO ≤ 2 h from nightly snapshot, ≤ 24 h full reindex worst case | rebuild in standby region | nightly snapshots exist purely to cut reindex time | | Crypto wallet metadata (xpubs, view keys, invoice index) | Secrets Manager + Postgres | RPO ≤ 5 min / RTO ≤ 60 min | replicated secrets; offline paper/SD backup of xpubs and Monero view key at 2 sites | watch-only by design — loss is an availability problem, never a fund-loss problem; spend keys never exist online (2-of-3 air-gapped multisig, RISK_REGISTER R-12) | | Chain node data (8 self-hosted chains) | EBS | RPO n/a (public data) / **RTO ≤ 6 h from weekly S3 chain snapshots** vs multi-day resync (Erigon, cardano-node) | restore snapshots in standby region | RISK_REGISTER R-13 | | Telemetry/metrics (TimescaleDB, Prometheus) | per-store | RPO 1 h / RTO best-effort | not replicated | operational, not product, data | ## 4. Backup cadence and verification | Cadence | Action | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Continuous | RDS WAL/PITR all domains; audit synchronous standby; S3 versioning + CRR on Tier-1 buckets | | Hourly | audit batch → S3 Object Lock | | Nightly | full RDS snapshots; Score/Envelope logical export; ES/Qdrant/Neo4j snapshots; backup-success report to SRE dashboard (a silent backup failure is a paged alert, not a log line) | | Weekly | chain-node data snapshots → S3 | | Monthly | **restore drill**: restore one domain DB (rotating through all domains ≈ once/year each) into an isolated VPC; verify with the domain's contract round-trip tests + row-count/checksum reconciliation; measure and record actual RTO vs target | | Quarterly | **region-loss game day** (§6); payments signing-station drill (RISK_REGISTER R-12); event-bus loss drill: drop the event-bus Redis node, verify outbox re-drive grants no duplicate entitlements (idempotency assertions from `V1/features.md:6169-6170`) | A backup that has not been restore-tested within its drill window is treated as nonexistent for go/no-go purposes. ## 5. Region-loss posture - **Per-data-plane primaries honoring residency** (`V1/ARCHITECTURE.md: 1225-1231`): US data plane primary `us-east-1`, warm standby `us-west-2`; EU data plane primary `eu-central-1`, warm standby `eu-west-1`. **DR never crosses a data plane** — EU-resident data is never restored into a US region; a residency-violating restore is a privacy incident, not a recovery. - **Warm standby contents:** Tier-1 DB replicas, replicated Tier-1 S3 buckets, IaC-applied but scaled-to-zero compute (Terraform under `infrastructure/terraform/` — ECS Fargate services at `desired_count = 0` cost nothing until promoted; `V1/DEPENDENCIES.md` §20), pre-pushed ECR images. - **GPU note:** RunPod capacity is region-flexible and AWS P5 reserve exists in the standby region's plan (RISK_REGISTER R-01); render-plane region loss re-routes workers rather than restoring state, because carry-state is node-portable (`V1/features.md:3728-3732`). - **Declared region-loss targets:** product readable/degraded ≤ 4 h (Tier-1 surfaces: auth, entitlements, Metis gateway, kept-artifact access); full service including render plane and search ≤ 24 h. Status page + outage copy paths are pre-written (`V1/features.md:6200-6202` runbook inventory). - **DNS/failover:** health-checked Route 53 failover for BFF and public viewer; manual decision (SRE Lead + Launch Director) to fail over the data plane — automatic cross-region DB promotion is explicitly out (split-brain risk outweighs the 4-h RTO). ## 6. Drill program and gate linkage - **DR-1 (pre-dogfood):** monthly restore drill executed once successfully on the metis and yemaya DBs (the two most contractually sensitive classes). - **DR-2 (pre-canary):** region-loss game day #1 — fail the US data plane to standby in a staging environment; pass = Tier-1 RTO ≤ 4 h measured, zero residency violations, entitlement outbox re-drive clean. This is load test LT-7 (CAPACITY_MODEL §7). - **DR-3 (pre-GA):** signing-station drill + event-bus loss drill passed; backup-verification dashboard green for 30 consecutive days. - GA go/no-go (LAUNCH_TIMELINE Phase 3→4) requires DR-1..DR-3 evidence attached; decision owner SRE Lead, co-sign Launch Director. ## 7. Ownership | Role | Owns | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | SRE Lead | SLO definitions/dashboards, error-budget policy, backup/restore program, region failover decision (with Launch Director) | | Platform/BFF Lead | BFF + event-bus SLIs, outbox implementations | | Realtime Lead (Psyche) | Psyche SLIs and degradation accounting | | Generation Platform Lead (Yemaya) | render-plane SLI, Score/Envelope backup class, determinism-based recompute path | | Metis Product Lead | Metis gateway SLO and institutional reporting of it | | Payments Lead | invoice-plane SLO, chain-freshness monitors, signing drills | | Privacy Lead | residency constraints on DR, restore-drill data handling |