Disciplines · Operations

On-Call

Platform on-call is a weekly rotation, primary + secondary.

8sections3 minread
Markdown source
On this page

Owner: platform team Last reviewed: 2026-05-28

How on-call works for Oshun V1. This is the master rotation doc; per-domain rotation specifics (lilith, iris, etc.) live in their domain runbook directories and reference this doc as the policy source.

Rotation shape#

Platform on-call is a weekly rotation, primary + secondary.

Tier Shift length Handoff Expected response
Primary Monday 09:00 → following Monday 09:00 (local) Monday standup <15 min for SEV-1, <30 min for SEV-2
Secondary Same week Monday standup Backstop the primary; takes over if primary unreachable >15 min

Per-domain rotations (lilith, iris, isis, aphrodite, etc.) follow the same shape but only page on alerts routed to their PagerDuty service (see alert-routing.md for the domain map). If a domain rotation goes unanswered for 30 min, the alert escalates to the platform primary.

What on-call does#

  • Acknowledge — Within 15 min of page for SEV-1, 30 min for SEV-2. Use the PagerDuty mobile app or the Slack /pagerduty ack command.
  • Triage — Open the linked runbook from the alert payload. Most runbooks have a "First 5 minutes" section; do that first.
  • Communicate — Post to #incidents with the status, the service affected, the runbook being followed, and the expected next update time. Update every 30 min until resolved.
  • Resolve or escalate — If the runbook doesn't fix it within the SLO breach window, escalate to the secondary; if both are blocked, escalate to the domain lead.
  • Postmortem — Every SEV-1 gets a postmortem within 5 business days; SEV-2 gets one only if it exceeded the SLO budget for the period.

Handoff checklist#

Mondays 09:00 local, the primary hands off to the next week's primary. The handoff covers:

  1. Any open incident (with ticket link).
  2. Any deploys in flight or scheduled this week.
  3. Any maintenance windows on the calendar.
  4. Anything noisy that you snoozed during your shift (so the next person doesn't get caught off-guard).
  5. Anything you'd want fixed before you'd take this shift again.

The handoff is logged in #oncall-handoff Slack channel for audit.

Escalation ladder#

  1. Primary — acknowledges and triages.
  2. Secondary — paged at the 15-min ack-failure boundary.
  3. Domain lead — paged by the secondary when the runbook doesn't apply or doesn't fix it within the SLO budget.
  4. Engineering lead — paged at the 60-min mark on SEV-1, or when leadership communication is needed.
  5. CTO / SVP eng — paged on extended SEV-1 (>2h), customer data exposure, or compliance escalation (PCI / GDPR Article 33-72h breach notification clock).

Tools#

  • PagerDuty — the source of truth for who's on-call and the paging contract.
  • Slack #incidents — incident comms (separate channel per active incident if the discussion is heavy).
  • Slack #alerts-prod — passive alert feed; the on-call primary should keep eyes on this.
  • GrafanaCloud — dashboards. The "On-Call" folder has the cross-domain overview dashboards.
  • Linear On-Call — capture follow-up tasks during the shift so they don't fall through the cracks.

When the on-call schedule needs to change#

  • Mid-shift swap: post in #oncall-handoff, update PagerDuty, done. No approval needed.
  • Permanent rotation change (new joiner, departure): open a PR against the rotation config in this doc + PagerDuty, get the team to ack.

Compensation / time off#

  • On-call shifts are compensated per the engineering compensation policy (separate doc, owned by people-ops).
  • Each on-call week earns a comp day to be taken within 90 days.
  • If you're on PTO or sick during your shift, swap with the secondary or another willing primary; document the swap in #oncall-handoff.

New-joiner shadow rotations#

  • New joiners shadow two full rotations as the secondary before taking primary.
  • The shadow week's primary writes a short retro in #oncall-handoff covering what the joiner did well and what to practice next time.