# On-Call

Owner: platform team Last reviewed: 2026-05-28

How on-call works for Oshun V1. This is the master rotation doc; per-domain
rotation specifics (lilith, iris, etc.) live in their domain runbook directories
and reference this doc as the policy source.

## Rotation shape

Platform on-call is a weekly rotation, primary + secondary.

| Tier      | Shift length                                  | Handoff        | Expected response                                               |
| --------- | --------------------------------------------- | -------------- | --------------------------------------------------------------- |
| Primary   | Monday 09:00 → following Monday 09:00 (local) | Monday standup | <15 min for SEV-1, <30 min for SEV-2                            |
| Secondary | Same week                                     | Monday standup | Backstop the primary; takes over if primary unreachable >15 min |

Per-domain rotations (lilith, iris, isis, aphrodite, etc.) follow the same shape
but only page on alerts routed to their PagerDuty service (see
[alert-routing.md](alert-routing.md) for the domain map). If a domain rotation
goes unanswered for 30 min, the alert escalates to the platform primary.

## What on-call does

- **Acknowledge** — Within 15 min of page for SEV-1, 30 min for SEV-2. Use the
  PagerDuty mobile app or the Slack `/pagerduty ack` command.
- **Triage** — Open the linked runbook from the alert payload. Most runbooks
  have a "First 5 minutes" section; do that first.
- **Communicate** — Post to `#incidents` with the status, the service affected,
  the runbook being followed, and the expected next update time. Update every 30
  min until resolved.
- **Resolve or escalate** — If the runbook doesn't fix it within the SLO breach
  window, escalate to the secondary; if both are blocked, escalate to the domain
  lead.
- **Postmortem** — Every SEV-1 gets a postmortem within 5 business days; SEV-2
  gets one only if it exceeded the SLO budget for the period.

## Handoff checklist

Mondays 09:00 local, the primary hands off to the next week's primary. The
handoff covers:

1. Any open incident (with ticket link).
2. Any deploys in flight or scheduled this week.
3. Any maintenance windows on the calendar.
4. Anything noisy that you snoozed during your shift (so the next person doesn't
   get caught off-guard).
5. Anything you'd want fixed before you'd take this shift again.

The handoff is logged in `#oncall-handoff` Slack channel for audit.

## Escalation ladder

1. **Primary** — acknowledges and triages.
2. **Secondary** — paged at the 15-min ack-failure boundary.
3. **Domain lead** — paged by the secondary when the runbook doesn't apply or
   doesn't fix it within the SLO budget.
4. **Engineering lead** — paged at the 60-min mark on SEV-1, or when leadership
   communication is needed.
5. **CTO / SVP eng** — paged on extended SEV-1 (>2h), customer data exposure, or
   compliance escalation (PCI / GDPR Article 33-72h breach notification clock).

## Tools

- **PagerDuty** — the source of truth for who's on-call and the paging contract.
- **Slack `#incidents`** — incident comms (separate channel per active incident
  if the discussion is heavy).
- **Slack `#alerts-prod`** — passive alert feed; the on-call primary should keep
  eyes on this.
- **GrafanaCloud** — dashboards. The "On-Call" folder has the cross-domain
  overview dashboards.
- **Linear `On-Call`** — capture follow-up tasks during the shift so they don't
  fall through the cracks.

## When the on-call schedule needs to change

- Mid-shift swap: post in `#oncall-handoff`, update PagerDuty, done. No approval
  needed.
- Permanent rotation change (new joiner, departure): open a PR against the
  rotation config in this doc + PagerDuty, get the team to ack.

## Compensation / time off

- On-call shifts are compensated per the engineering compensation policy
  (separate doc, owned by people-ops).
- Each on-call week earns a comp day to be taken within 90 days.
- If you're on PTO or sick during your shift, swap with the secondary or another
  willing primary; document the swap in `#oncall-handoff`.

## New-joiner shadow rotations

- New joiners shadow two full rotations as the secondary before taking primary.
- The shadow week's primary writes a short retro in `#oncall-handoff` covering
  what the joiner did well and what to practice next time.