Owner: platform team Last reviewed: 2026-05-28
Production SLOs for the V1 services. Each row gates the alert routing in alert-routing.md and the budget calculations in the Grafana "SLO" dashboards.
This doc is policy. Per-service implementation lives in:
infra/monitoring/<domain>/grafana/dashboards/*-slo.json— Grafana SLO panel definitions.infra/monitoring/<domain>/prometheus/alerts.yml— alerting rules (latency, error rate).libs/shared/observability— instrumentation helpers.
Eve's cross-service assistant path additionally uses the source-verified,
per-plane contract in
EVE_SOTA_RELIABILITY_SLO_CONTRACT_2026-09.md.
That contract is stricter where a service average could conceal failed tasks,
late or duplicate effects, missing cost, grounding failure, or a data-boundary
violation. Missing plane telemetry is UNKNOWN, never an availability pass.
SLO budget rules#
- Each SLO has an error budget =
(1 − target) × measurement window. - We measure on a 30-day rolling window.
- If a service burns >50% of its budget in a single week, the domain freezes non-critical deploys until the budget is restored.
- Budget burns trigger a postmortem (on-call.md §Postmortem).
Targets#
User-facing surfaces#
| Service | Availability | Latency p95 | Latency p99 | Notes |
|---|---|---|---|---|
| oshun-shell | 99.95% | 200 ms (TTFB) | 500 ms | Front-door shell; outage = full product outage |
| iris/api | 99.9% | 400 ms | 1.0 s | Assistant chat — most critical user surface |
| lilith/svc-conversation | 99.9% | 250 ms (text) | 600 ms | WebSocket message processing |
| aphrodite/payment | 99.95% | 600 ms | 1.5 s | Checkout flow; budget = revenue |
| aphrodite/streaming | 99.5% | n/a (streaming) | n/a | RTMP availability; viewer-side latency in aphrodite-streaming-viewer-slo.json |
| nyx/api | 99.5% | 200 ms | 500 ms | Astronomy data API; mostly cacheable |
| tara/api | 99.5% | 300 ms | 750 ms | Meditation session APIs |
| veritas/api | 99.5% | 500 ms | 1.5 s | News/content; high read volume |
| yemaya/api | 99.9% | 300 ms | 800 ms | Identity / tenant / settings |
| metis/api-gateway | 99.95% | 50 ms (forwarding overhead) | 150 ms | Edge gateway |
Internal / async surfaces#
| Service | Availability | Job p95 | Job p99 | Notes |
|---|---|---|---|---|
| lilith/svc-ai (inference) | 99.5% | 4 s | 10 s | Model inference; latency varies by model |
| lilith/svc-tts | 99.5% | 2 s | 5 s | TTS synthesis |
| isis/generation-api | 99.5% | 30 s | 90 s | Image / video generation |
| veritas/ingestion | 99.5% | 60 s | 300 s | RSS / sitemap collectors |
| veritas/video | 99.5% | 5 min | 15 min | Video generation pipeline |
| yemaya/workers | 99.5% | 30 s | 120 s | Notification / email workers |
| oshun/bff (worker mode) | 99.9% | 1 s | 3 s | Cross-domain orchestration |
Data / storage surfaces#
| Service | Availability | Notes |
|---|---|---|
| Postgres primary (per domain) | 99.95% | RDS Multi-AZ; managed by AWS, but app-level health check counts |
| Redis (per domain) | 99.9% | ElastiCache with replica fail-over |
| MinIO / S3 (uploads) | 99.99% | Vendor SLA; app-level GET-success counts |
| Kafka (event bus) | 99.9% | When enabled per profile |
How we measure#
- Availability —
(successful_requests / total_requests)where successful = HTTP 2xx-4xx (4xx is a client error, not a service failure). Computed from Prometheusrate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]). - Latency — Prometheus
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))for p95;0.99for p99. - Job duration — Worker tasks emit a custom histogram
oshun_job_duration_secondswith the job-type label.
Excluded events#
We exclude from SLO accounting:
- Synthetic monitoring traffic.
- Health-check pings (
/health,/healthz). - Internal admin endpoints (
/v1/admin/*). - Scheduled maintenance windows (set via the maintenance toggle in PagerDuty).
Reviewing targets#
The SLO doc is reviewed quarterly. Each domain owner brings:
- Prior-quarter actuals (uptime, p95/p99) from Grafana.
- Burn-rate alert frequency.
- Whether the target was hit, missed by <10%, or missed by >10%.
- Proposed target adjustments (with rationale).
Permanent target changes require sign-off from the domain lead and the platform team. Tightening targets is encouraged when the service has been comfortably within them for 2+ quarters; loosening targets requires evidence of architectural limits and a plan to re-tighten.