Status: planning baseline, created 2026-06-12 to close
V1_V7_PLAN_SET_AUDIT_2026-06-12.md §6.2 (V1: "no availability SLOs (latency
budgets exist, uptime budgets don't)") and §6.1(6) (no DR/backup objectives).
Latency budgets cited below are the spec's own; the availability targets and
RPO/RTO figures are planning assumptions adopted 2026-06-12, each with its
derivation, to be ratified at the GA go/no-go.
1. Availability SLOs per surface (monthly windows)#
Tiering principle: surfaces that gate money, identity, safety, or
institutional contracts get 99.9; realtime/generative surfaces get 99.5 with
honest degraded modes; the GPU render plane gets 99.0 because the spec already
defines a degradation ladder that preserves the session
(V1/features.md:4420-4422) — availability money is better spent on the ladder
than on gold-plating the render plane.
| Surface | SLI definition | Target | Monthly error budget |
|---|---|---|---|
BFF API (apps/oshun/bff) |
non-5xx responses ÷ valid requests; latency SLI: p99 ≤ 800 ms reads / 1.5 s writes | 99.9 % | 43 min 49 s |
| Auth/session issuance | successful token issuance ÷ valid attempts | 99.9 % | 43 min 49 s |
| Psyche realtime — platform | session-minutes in which a session could be established or continued in any mode (text counts — the fallback chain is the design, V1/features.md:2239-2246) |
99.5 % | 3 h 39 min |
| Psyche realtime — voice mode | session-minutes with voice available within budget (degradation to text counts against this SLI but not the platform SLI) | 99.0 % | 7 h 18 min |
| Living Scenes control plane (Conductor session establishment, score CRUD, cue gate) | successful session starts + cue acks ÷ attempts | 99.5 % | 3 h 39 min |
| Living Scenes render plane | session-minutes not in BLACK queue state (CAPACITY_MODEL §6.2); first-frame budget compliance ≥ 95 % of sessions | 99.0 % | 7 h 18 min |
Metis gateway (apps/metis/api-gateway; LTI launches, gradebook emission) |
non-5xx ÷ valid; LTI launch success ratio | 99.9 % (institutional contracts quote this number) | 43 min 49 s |
Public viewer / attestation pages (oshun.app/scene/<id>) |
successful page render ÷ requests (CDN + static-first design) | 99.95 % | 21 min 54 s |
| Payments invoice plane (invoice issuance + payment-detection pipeline) | invoice issuance success; per-chain detection freshness: chain tip lag ≤ 2 blocks for 99 % of minutes (RISK_REGISTER R-13 monitor) | 99.9 % | 43 min 49 s |
Event bus delivery (@oshun/event-bus — Redis pub/sub with TTL-keyed replay) |
events delivered-at-least-once within 60 s ÷ published | 99.9 % | 43 min 49 s |
Notes:
- Gradebook emission is asynchronous with idempotent re-emission
(
V1/features.md:1569-1572) — its SLO is freshness (99 % of entries emitted ≤ 15 min after trigger), carried under the Metis gateway row. - Crisis-path availability is not an SLO with a budget: the crisis frame and cue kill path fail closed and have reserved capacity (CAPACITY_MODEL §6.2); any crisis-path unavailability is a Sev-1 regardless of duration.
2. Error-budget policy#
- Burn-rate alerts per the multiwindow standard: page at 14.4× budget burn over 1 h, ticket at 6× over 6 h (both windows must agree).
- Budget exhausted → feature freeze on that surface: only reliability/rollback changes merge until the 30-day window recovers; canary promotions (LAUNCH_TIMELINE Phase 3) are blocked while any RED-tier surface is over budget.
- SLO review monthly; targets ratchet only after two consecutive months of ≥ 50 % unspent budget.
3. DR objectives per data class#
Region posture context (§5) — per-data-plane primary/standby. "RPO" is data loss tolerance; "RTO" time-to-restore service on that class.
| Data class | Store | Failure-domain RPO/RTO (AZ or instance loss) | Region-loss RPO/RTO | Rationale |
|---|---|---|---|---|
Per-domain OLTP (tara, nisaba, arete, nyx, veritas, metis, lilith, isis, sophia, hathor, bellona; V1/ARCHITECTURE.md:1195-1202) |
RDS Postgres multi-AZ, PITR | RPO ≤ 5 min / RTO ≤ 60 min (multi-AZ failover itself ≤ 5 min) | RPO ≤ 15 min / RTO ≤ 4 h (cross-region replica for Tier-1 DBs: identity/entitlements, audit, metis; snapshot-restore for the rest, RTO ≤ 12 h) | 5-min WAL shipping is RDS-native; Tier-1 split keeps cross-region replica cost on the records that are contractually or financially irreplaceable |
Audit platform (@oshun/audit-platform, append-only) |
Postgres + hourly export | RPO ≤ 1 min (synchronous standby) / RTO ≤ 60 min | RPO ≤ 60 min via hourly batch export to S3 Object Lock (WORM) | audit is the substrate for DSAR, FERPA access, payment disputes — near-zero loss tolerance, and WORM defends against the erase-your-tracks attacker |
Kept-artifact Scores + Render Envelopes + Cue Plans (source of truth, kilobytes each, V1/features.md:4195-4199) |
Postgres (yemaya) + nightly export | RPO ≤ 5 min / RTO ≤ 60 min | RPO ≤ 15 min / RTO ≤ 4 h (Tier-1) | "offerings persist for the lifetime of the user account" (V1/features.md:4206-4208) makes these irreplaceable; the MP4s are not |
| Materialized MP4 caches + share renders | S3 (MinIO in dev) | RPO 24 h / RTO = re-render on demand | same | re-renderable bit-equivalently from Score + Envelope by the determinism gate (V1/features.md:4193-4199) — DR by recomputation is the designed path |
| Source media, ingestion originals, model artifacts | S3, versioned | RPO ≤ 15 min (S3 versioning + same-region durability) / RTO ≤ 2 h | cross-region replication on sources and provenance buckets, RPO ≤ 15 min |
originals are not recomputable |
| Redis — cache tier | ElastiCache | RPO = n/a (rebuild) / RTO ≤ 15 min warm-up | rebuild | cache by definition |
Redis — event bus (@oshun/event-bus) |
ElastiCache, AOF everysec | RPO ≤ 1 s–1 min / RTO ≤ 30 min | not replicated cross-region — gap re-driven from the transactional outbox | consumers are idempotent with dedup/replay required by the test matrix (V1/features.md:6169-6170); critical producers (payments-bridge entitlement grants, gradebook emission) write an outbox row in their domain Postgres in the same transaction, so the bus is never the only copy of a money- or grade-bearing event |
Search/vector/graph indexes (Elasticsearch, Qdrant, Neo4j; V1/ARCHITECTURE.md:1213-1219) |
managed/EC2 | RPO n/a (derived data) / RTO ≤ 2 h from nightly snapshot, ≤ 24 h full reindex worst case | rebuild in standby region | nightly snapshots exist purely to cut reindex time |
| Crypto wallet metadata (xpubs, view keys, invoice index) | Secrets Manager + Postgres | RPO ≤ 5 min / RTO ≤ 60 min | replicated secrets; offline paper/SD backup of xpubs and Monero view key at 2 sites | watch-only by design — loss is an availability problem, never a fund-loss problem; spend keys never exist online (2-of-3 air-gapped multisig, RISK_REGISTER R-12) |
| Chain node data (8 self-hosted chains) | EBS | RPO n/a (public data) / RTO ≤ 6 h from weekly S3 chain snapshots vs multi-day resync (Erigon, cardano-node) | restore snapshots in standby region | RISK_REGISTER R-13 |
| Telemetry/metrics (TimescaleDB, Prometheus) | per-store | RPO 1 h / RTO best-effort | not replicated | operational, not product, data |
4. Backup cadence and verification#
| Cadence | Action |
|---|---|
| Continuous | RDS WAL/PITR all domains; audit synchronous standby; S3 versioning + CRR on Tier-1 buckets |
| Hourly | audit batch → S3 Object Lock |
| Nightly | full RDS snapshots; Score/Envelope logical export; ES/Qdrant/Neo4j snapshots; backup-success report to SRE dashboard (a silent backup failure is a paged alert, not a log line) |
| Weekly | chain-node data snapshots → S3 |
| Monthly | restore drill: restore one domain DB (rotating through all domains ≈ once/year each) into an isolated VPC; verify with the domain's contract round-trip tests + row-count/checksum reconciliation; measure and record actual RTO vs target |
| Quarterly | region-loss game day (§6); payments signing-station drill (RISK_REGISTER R-12); event-bus loss drill: drop the event-bus Redis node, verify outbox re-drive grants no duplicate entitlements (idempotency assertions from V1/features.md:6169-6170) |
A backup that has not been restore-tested within its drill window is treated as nonexistent for go/no-go purposes.
5. Region-loss posture#
- Per-data-plane primaries honoring residency
(
V1/ARCHITECTURE.md: 1225-1231): US data plane primaryus-east-1, warm standbyus-west-2; EU data plane primaryeu-central-1, warm standbyeu-west-1. DR never crosses a data plane — EU-resident data is never restored into a US region; a residency-violating restore is a privacy incident, not a recovery. - Warm standby contents: Tier-1 DB replicas, replicated Tier-1 S3 buckets,
IaC-applied but scaled-to-zero compute (Terraform under
infrastructure/terraform/— ECS Fargate services atdesired_count = 0cost nothing until promoted;V1/DEPENDENCIES.md§20), pre-pushed ECR images. - GPU note: RunPod capacity is region-flexible and AWS P5 reserve exists in
the standby region's plan (RISK_REGISTER R-01); render-plane region loss
re-routes workers rather than restoring state, because carry-state is
node-portable (
V1/features.md:3728-3732). - Declared region-loss targets: product readable/degraded ≤ 4 h (Tier-1
surfaces: auth, entitlements, Metis gateway, kept-artifact access); full
service including render plane and search ≤ 24 h. Status page + outage copy
paths are pre-written (
V1/features.md:6200-6202runbook inventory). - DNS/failover: health-checked Route 53 failover for BFF and public viewer; manual decision (SRE Lead + Launch Director) to fail over the data plane — automatic cross-region DB promotion is explicitly out (split-brain risk outweighs the 4-h RTO).
6. Drill program and gate linkage#
- DR-1 (pre-dogfood): monthly restore drill executed once successfully on the metis and yemaya DBs (the two most contractually sensitive classes).
- DR-2 (pre-canary): region-loss game day #1 — fail the US data plane to standby in a staging environment; pass = Tier-1 RTO ≤ 4 h measured, zero residency violations, entitlement outbox re-drive clean. This is load test LT-7 (CAPACITY_MODEL §7).
- DR-3 (pre-GA): signing-station drill + event-bus loss drill passed; backup-verification dashboard green for 30 consecutive days.
- GA go/no-go (LAUNCH_TIMELINE Phase 3→4) requires DR-1..DR-3 evidence attached; decision owner SRE Lead, co-sign Launch Director.
7. Ownership#
| Role | Owns |
|---|---|
| SRE Lead | SLO definitions/dashboards, error-budget policy, backup/restore program, region failover decision (with Launch Director) |
| Platform/BFF Lead | BFF + event-bus SLIs, outbox implementations |
| Realtime Lead (Psyche) | Psyche SLIs and degradation accounting |
| Generation Platform Lead (Yemaya) | render-plane SLI, Score/Envelope backup class, determinism-based recompute path |
| Metis Product Lead | Metis gateway SLO and institutional reporting of it |
| Payments Lead | invoice-plane SLO, chain-freshness monitors, signing drills |
| Privacy Lead | residency constraints on DR, restore-drill data handling |