# V3 SLOs and Error Budgets

Plan id: `v3-slo-error-budgets.v1`

Verifier: `pnpm verify:v3 slo-error-budgets` (proposed)

Evidence artifacts: `V3/operations/slo-monthly-report-<YYYY-MM>.json`
(proposed), Grafana folder `oshun-v3-launch-readiness`, board
`v3-slo-error-budgets` (proposed)

Owner: V3 Release Captain (budget policy), per-SLO owners below (operations)

The validation gates (`COLD_JOIN_VALIDATION.md`,
`PIXEL_STREAMING_SESSION_START_SUCCESS.md`, `VOICE_LATENCY_VALIDATION.md`,
`FLEET_LOAD_VALIDATION.md`, `RECONNECT_VALIDATION.md`,
`PIXEL_STREAMING_RTT_VALIDATION.md`) prove the numbers were met **once, before
GA**. This document converts the user-critical subset into ongoing monthly SLOs
with error budgets, burn-rate alerting, and a freeze policy — the mechanism that
keeps the numbers true in month two.

## SLO Catalog

Window: calendar month, all SLOs. SLIs are measured from production telemetry
(`TELEMETRY_SCHEMA_V3.md` events), not synthetic probes, except availability
which combines both.

| #   | SLI (good-event definition)                                                                                                           | Source gate                  | Monthly SLO | Error budget (bad events)                               | Owner (on-call)             |
| --- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ----------- | ------------------------------------------------------- | --------------------------- |
| 1   | Pixel Streaming session-start attempts that reach first frame                                                                         | session-start success ≥99.5% | 99.5%       | 0.5% of attempts                                        | `oshun-capacity-ops-oncall` |
| 2   | Pxstream sessions with first frame ≤ 8 s                                                                                              | first-frame p99 ≤8 s         | 99.0%       | 1% of sessions                                          | `oshun-capacity-ops-oncall` |
| 3   | Pxstream RTT samples with end-to-end ≤ 80 ms                                                                                          | RTT p99 ≤80 ms               | 99.0%       | 1% of samples                                           | `oshun-capacity-ops-oncall` |
| 4   | Voice samples with mouth-to-ear ≤ 80 ms                                                                                               | voice p95 ≤80 ms             | 95.0%       | 5% of samples                                           | `oshun-voice-ops-oncall`    |
| 5   | Voice packets delivered (1 − loss rate)                                                                                               | voice loss ≤0.5%             | 99.5%       | 0.5% of packets                                         | `oshun-voice-ops-oncall`    |
| 6   | Cold joins fully rendered ≤ 5 s                                                                                                       | cold-join p95 ≤5 s           | 95.0%       | 5% of cold joins                                        | `oshun-realtime-ops-oncall` |
| 7   | Transient reconnects resumed ≤ 3 s with state preserved                                                                               | reconnect p95 ≤3 s           | 95.0%       | 5% of reconnect attempts                                | `oshun-realtime-ops-oncall` |
| 8   | BFF auth + launch-resolver requests answered non-5xx ≤ 1 s (availability SLO — the gate set had latency budgets but no uptime budget) | new                          | 99.9%       | 0.1% of requests (~43 min/month at steady request rate) | `oshun-realtime-ops-oncall` |

Notes on target choice: where a gate asserted a p95 budget, the SLO is the same
threshold restated as "95% of events good" (identical claim, now continuously
measured); p99 gates become 99% SLOs. The session-start and voice-loss targets
carry over numerically (99.5%). Targets are ratchet-protected: lowering any SLO
target requires the same release-captain sign-off as weakening a release gate.

## Burn-Rate Alerting

Standard multiwindow multi-burn-rate policy, identical for every SLO. Burn rate
= (observed bad-event ratio) ÷ (budget ratio); burn rate 1 = exactly spending
the budget over the month.

| Alert     | Condition                                           | Budget consumed when it fires | Action                |
| --------- | --------------------------------------------------- | ----------------------------- | --------------------- |
| Fast burn | burn ≥ 14.4 over 1 h AND ≥ 14.4 over the last 5 min | 2% of monthly budget in 1 h   | Page the SLO owner    |
| Slow burn | burn ≥ 6 over 6 h AND ≥ 6 over the last 30 min      | 5% of monthly budget in 6 h   | Page the SLO owner    |
| Trickle   | burn ≥ 1 over 3 days                                | 10% in 3 days                 | Ticket, weekly review |

Concrete thresholds the dashboards encode (bad-event ratio that trips fast
burn): SLO 99.9% → 1.44%; SLO 99.5% → 7.2%; SLO 99.0% → 14.4%; SLO 95.0% → 72%
(clamped to 50% — a majority-bad signal pages regardless of arithmetic). The
two-window AND prevents paging on a recovered spike; the 5-min/30-min short
windows make alerts reset promptly after recovery.

Burn alerts route through the same PagerDuty services as the existing Grafana
boards (`GRAFANA_DASHBOARDS.md`); the `v3-slo-error-budgets` board publishes,
per SLO: remaining budget, 1 h / 6 h / 3 d burn rates, and month-to-date trend.

## Budget Accounting Rules

- Planned maintenance is **not** excluded — users do not care why. Schedule
  maintenance against budget headroom.
- Events during a declared SEV-1/SEV-2 incident still count; the incident review
  must reconcile budget spend with the postmortem.
- Synthetic-probe gaps (monitoring outage) count as bad for SLO 8 and are
  excluded for SLOs 1–7 (telemetry-sourced); a telemetry outage > 1 h is itself
  a paging condition because it blinds the budget.
- Per-POP/per-region budgets: SLOs 1–3 are also tracked per POP; a single POP
  burning > 3x while the global SLO is green pages the capacity on-call
  (regional users experience the regional number, not the global average).

## Freeze Policy

Enforced by the V3 Release Captain through the existing
`release-readiness-decision` operator action — a freeze revokes rollout tokens
for the affected services.

| Budget state (per SLO, month to date) | Policy                                                                                                                                                                                     |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| < 50% consumed                        | Normal releases.                                                                                                                                                                           |
| ≥ 50% consumed before day 20          | Heightened review: canary bake times double; risky changes (new features touching the SLO's service path) need explicit owner sign-off.                                                    |
| 100% consumed (budget exhausted)      | **Feature freeze** on every service in that SLO's delivery path: only reliability fixes, rollbacks, and security patches ship. New-feature work continues on branches but does not deploy. |
| Exhausted + still burning ≥ 1x        | Freeze plus mandatory incident: commander assigned, daily exec update, scope-shedding options (backpressure, queue, regional throttles) on the table.                                      |

Freeze exit: 7 consecutive days with burn rate < 1 AND the postmortem actions
for the dominant burn causes merged. Exiting freeze is a logged release-captain
decision (`operator.slo.freeze_lifted`), not an automatic flip.

Override: shipping a feature during freeze requires joint release-captain +
engineering-lead sign-off recorded in the Operator Console with the reason — the
same escape-valve discipline as every other gate, never silent.

## Status Page Tie-In

The public status page (`/status`, per `MARKETING_PUBLIC_WEB.md`) lists the V3
launch router, world server, Pixel Streaming relay, and fallback web client and
exposes `data-v3-health-status`:

- Fast-burn alert active on SLOs 1, 6, or 8 (the join-path SLOs) → the affected
  component shows `degraded`; `data-v3-health-status` leaves `operational`.
- Budget exhausted on any SLO → component shows `degraded` for the remainder of
  remediation with an honest note ("elevated session-start failures"); freeze
  state itself is internal, but its user-visible cause is not hidden.
- Status copy changes follow the `incident-communications` action rules: amend,
  never delete.

## Monthly Review

First business day of each month, owned by the release captain, attended by
every SLO owner: previous month's budget spend per SLO, burn-event inventory,
threshold drift (did we alert too often / too late), and whether any SLO target
should tighten (loosening goes through the ratchet rule). Output is the
`slo-monthly-report-<YYYY-MM>.json` evidence file plus updates to this document
by PR.

## Fail-Closed Criteria

This gate fails closed when any of the following holds:

- any catalog SLO lacks a live SLI series on the `v3-slo-error-budgets` board, a
  named owner, or wired fast/slow burn alerts (verified by injecting a synthetic
  bad-event stream in staging and observing the page — evidence
  `drill-slo-burn-alerts-ga`);
- an SLO target is lowered without a recorded release-captain decision;
- the freeze policy is not wired to the `release-readiness-decision` action (a
  freeze that cannot revoke rollout tokens is advisory, not a policy);
- the status-page component mapping is missing for any join-path SLO;
- a monthly report is more than 10 days late;
- telemetry powering SLOs 1–7 deviates from `TELEMETRY_SCHEMA_V3.md` without a
  schema-version bump (silent SLI redefinition is the SLO equivalent of a
  fabricated result).