V3 SLOs and Error Budgets — Lilith Metaverse

Plan id: v3-slo-error-budgets.v1

Verifier: pnpm verify:v3 slo-error-budgets (proposed)

Evidence artifacts: V3/operations/slo-monthly-report-<YYYY-MM>.json (proposed), Grafana folder oshun-v3-launch-readiness, board v3-slo-error-budgets (proposed)

Owner: V3 Release Captain (budget policy), per-SLO owners below (operations)

The validation gates (COLD_JOIN_VALIDATION.md, PIXEL_STREAMING_SESSION_START_SUCCESS.md, VOICE_LATENCY_VALIDATION.md, FLEET_LOAD_VALIDATION.md, RECONNECT_VALIDATION.md, PIXEL_STREAMING_RTT_VALIDATION.md) prove the numbers were met once, before GA. This document converts the user-critical subset into ongoing monthly SLOs with error budgets, burn-rate alerting, and a freeze policy — the mechanism that keeps the numbers true in month two.

SLO Catalog#

Window: calendar month, all SLOs. SLIs are measured from production telemetry (TELEMETRY_SCHEMA_V3.md events), not synthetic probes, except availability which combines both.

#	SLI (good-event definition)	Source gate	Monthly SLO	Error budget (bad events)	Owner (on-call)
1	Pixel Streaming session-start attempts that reach first frame	session-start success ≥99.5%	99.5%	0.5% of attempts	`oshun-capacity-ops-oncall`
2	Pxstream sessions with first frame ≤ 8 s	first-frame p99 ≤8 s	99.0%	1% of sessions	`oshun-capacity-ops-oncall`
3	Pxstream RTT samples with end-to-end ≤ 80 ms	RTT p99 ≤80 ms	99.0%	1% of samples	`oshun-capacity-ops-oncall`
4	Voice samples with mouth-to-ear ≤ 80 ms	voice p95 ≤80 ms	95.0%	5% of samples	`oshun-voice-ops-oncall`
5	Voice packets delivered (1 − loss rate)	voice loss ≤0.5%	99.5%	0.5% of packets	`oshun-voice-ops-oncall`
6	Cold joins fully rendered ≤ 5 s	cold-join p95 ≤5 s	95.0%	5% of cold joins	`oshun-realtime-ops-oncall`
7	Transient reconnects resumed ≤ 3 s with state preserved	reconnect p95 ≤3 s	95.0%	5% of reconnect attempts	`oshun-realtime-ops-oncall`
8	BFF auth + launch-resolver requests answered non-5xx ≤ 1 s (availability SLO — the gate set had latency budgets but no uptime budget)	new	99.9%	0.1% of requests (~43 min/month at steady request rate)	`oshun-realtime-ops-oncall`

Notes on target choice: where a gate asserted a p95 budget, the SLO is the same threshold restated as "95% of events good" (identical claim, now continuously measured); p99 gates become 99% SLOs. The session-start and voice-loss targets carry over numerically (99.5%). Targets are ratchet-protected: lowering any SLO target requires the same release-captain sign-off as weakening a release gate.

Burn-Rate Alerting#

Standard multiwindow multi-burn-rate policy, identical for every SLO. Burn rate = (observed bad-event ratio) ÷ (budget ratio); burn rate 1 = exactly spending the budget over the month.

Alert	Condition	Budget consumed when it fires	Action
Fast burn	burn ≥ 14.4 over 1 h AND ≥ 14.4 over the last 5 min	2% of monthly budget in 1 h	Page the SLO owner
Slow burn	burn ≥ 6 over 6 h AND ≥ 6 over the last 30 min	5% of monthly budget in 6 h	Page the SLO owner
Trickle	burn ≥ 1 over 3 days	10% in 3 days	Ticket, weekly review

Concrete thresholds the dashboards encode (bad-event ratio that trips fast burn): SLO 99.9% → 1.44%; SLO 99.5% → 7.2%; SLO 99.0% → 14.4%; SLO 95.0% → 72% (clamped to 50% — a majority-bad signal pages regardless of arithmetic). The two-window AND prevents paging on a recovered spike; the 5-min/30-min short windows make alerts reset promptly after recovery.

Burn alerts route through the same PagerDuty services as the existing Grafana boards (GRAFANA_DASHBOARDS.md); the v3-slo-error-budgets board publishes, per SLO: remaining budget, 1 h / 6 h / 3 d burn rates, and month-to-date trend.

Budget Accounting Rules#

Planned maintenance is not excluded — users do not care why. Schedule maintenance against budget headroom.
Events during a declared SEV-1/SEV-2 incident still count; the incident review must reconcile budget spend with the postmortem.
Synthetic-probe gaps (monitoring outage) count as bad for SLO 8 and are excluded for SLOs 1–7 (telemetry-sourced); a telemetry outage > 1 h is itself a paging condition because it blinds the budget.
Per-POP/per-region budgets: SLOs 1–3 are also tracked per POP; a single POP burning > 3x while the global SLO is green pages the capacity on-call (regional users experience the regional number, not the global average).

Freeze Policy#

Enforced by the V3 Release Captain through the existing release-readiness-decision operator action — a freeze revokes rollout tokens for the affected services.

Budget state (per SLO, month to date)	Policy
< 50% consumed	Normal releases.
≥ 50% consumed before day 20	Heightened review: canary bake times double; risky changes (new features touching the SLO's service path) need explicit owner sign-off.
100% consumed (budget exhausted)	Feature freeze on every service in that SLO's delivery path: only reliability fixes, rollbacks, and security patches ship. New-feature work continues on branches but does not deploy.
Exhausted + still burning ≥ 1x	Freeze plus mandatory incident: commander assigned, daily exec update, scope-shedding options (backpressure, queue, regional throttles) on the table.

Freeze exit: 7 consecutive days with burn rate < 1 AND the postmortem actions for the dominant burn causes merged. Exiting freeze is a logged release-captain decision (operator.slo.freeze_lifted), not an automatic flip.

Override: shipping a feature during freeze requires joint release-captain + engineering-lead sign-off recorded in the Operator Console with the reason — the same escape-valve discipline as every other gate, never silent.

Status Page Tie-In#

The public status page (/status, per MARKETING_PUBLIC_WEB.md) lists the V3 launch router, world server, Pixel Streaming relay, and fallback web client and exposes data-v3-health-status:

Fast-burn alert active on SLOs 1, 6, or 8 (the join-path SLOs) → the affected component shows degraded; data-v3-health-status leaves operational.
Budget exhausted on any SLO → component shows degraded for the remainder of remediation with an honest note ("elevated session-start failures"); freeze state itself is internal, but its user-visible cause is not hidden.
Status copy changes follow the incident-communications action rules: amend, never delete.

Monthly Review#

First business day of each month, owned by the release captain, attended by every SLO owner: previous month's budget spend per SLO, burn-event inventory, threshold drift (did we alert too often / too late), and whether any SLO target should tighten (loosening goes through the ratchet rule). Output is the slo-monthly-report-<YYYY-MM>.json evidence file plus updates to this document by PR.

Fail-Closed Criteria#

This gate fails closed when any of the following holds:

any catalog SLO lacks a live SLI series on the v3-slo-error-budgets board, a named owner, or wired fast/slow burn alerts (verified by injecting a synthetic bad-event stream in staging and observing the page — evidence drill-slo-burn-alerts-ga);
an SLO target is lowered without a recorded release-captain decision;
the freeze policy is not wired to the release-readiness-decision action (a freeze that cannot revoke rollout tokens is advisory, not a policy);
the status-page component mapping is missing for any join-path SLO;
a monthly report is more than 10 days late;
telemetry powering SLOs 1–7 deviates from TELEMETRY_SCHEMA_V3.md without a schema-version bump (silent SLI redefinition is the SLO equivalent of a fabricated result).