Eve library

phase-152.md

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

Phase 152: Sovereign Incident Management & On-Call Platform — PagerDuty / Opsgenie / VictorOps (Splunk On-Call) / xMatters / FireHydrant / incident.io / Rootly / Blameless / Grafana OnCall / Better Stack Replacement#

Closes the incident-management sovereignty gap. Phase 137 briefly mentions "PagerDuty / sovereign" for live-ops. Phase 149 (Crash Analytics) and Phase 151 (EDR) both fire alerts but neither manages schedules, escalations, SLAs, stakeholder comms, post-mortems, or retros. This phase delivers an end-to-end incident platform for Oshun and external customers.

Sovereign replacement for PagerDuty, Opsgenie, Splunk On-Call (VictorOps), xMatters, FireHydrant, incident.io, Rootly, Blameless, Jeli, Grafana OnCall, Better Stack Uptime / Incidents, statuspage.io, StatusGator, ilert, and Squadcast.


152.1 @neith/incident-core — Incident Model & Lifecycle#

  • 152.1.1 Incident entity (id / title / severity / services / owner)
  • 152.1.2 Severity scale SEV-1 → SEV-5 with auto-bump rules
  • 152.1.3 Incident lifecycle (triggered / investigating / identified / mitigating / resolved)
  • 152.1.4 Status transitions with required fields
  • 152.1.5 Services / components taxonomy
  • 152.1.6 Dependency graph (upstream / downstream blast-radius)
  • 152.1.7 Incident linking (parent / child / related)
  • 152.1.8 Incident merge + split with full audit
  • 152.1.9 Custom fields per-team
  • 152.1.10 Tags + saved filters
  • 152.1.11 Timeline with automatic + manual entries
  • 152.1.12 Impact estimation (users / revenue / SLO burn)
  • 152.1.13 Incident closeout checklist
  • 152.1.14 Postmortem doc auto-scaffold on resolve
  • 152.1.15 Retention + anonymisation for long-term storage

152.2 @neith/oncall — On-Call Schedules & Rotations#

  • 152.2.1 Weekly / daily / follow-the-sun rotation templates
  • 152.2.2 Layered schedule (primary / secondary / manager)
  • 152.2.3 Override window with audit trail
  • 152.2.4 Holiday + PTO calendar import (iCal / Google / sovereign cal)
  • 152.2.5 Shift swap request + approval
  • 152.2.6 On-call time-budget report (hours per person per period)
  • 152.2.7 Fair-distribution scheduler suggestions
  • 152.2.8 Regional / timezone aware routing
  • 152.2.9 Team rotation + escalation matrix composer
  • 152.2.10 Current + upcoming on-call view (who to page now)
  • 152.2.11 "Who is on-call" API for other tools
  • 152.2.12 Cross-team pager hand-off
  • 152.2.13 Burnout detection heuristic (nights / weekends / page count)
  • 152.2.14 Vacation embargo windows
  • 152.2.15 Trainee / shadow mode (dual-page with senior)

152.3 @neith/escalation — Escalation Policies & Routing#

  • 152.3.1 Policy DSL (step → notify → wait → escalate)
  • 152.3.2 Round-robin + all-at-once + parallel modes
  • 152.3.3 Ack timer per step (configurable)
  • 152.3.4 Escalate on no-ack → next level
  • 152.3.5 Escalate on reject / unavailable
  • 152.3.6 Conditional routing (service / env / time-of-day / severity)
  • 152.3.7 Team-based routing (CODEOWNERS / service catalog)
  • 152.3.8 Role-based routing (incident commander, scribe, SME)
  • 152.3.9 Fallback to manager tree
  • 152.3.10 Loop prevention / max-cycle guard

152.4 @neith/alert-ingest — Alert Ingestion & Routing#

  • 152.4.1 Generic webhook endpoint with signature verify
  • 152.4.2 Native integration: Phase 149 (crash)
  • 152.4.3 Native integration: Phase 151 (EDR)
  • 152.4.4 Native integration: Phase 98 (Nous observability)
  • 152.4.5 Prometheus Alertmanager sink
  • 152.4.6 Grafana alert sink
  • 152.4.7 Datadog / New Relic / Splunk compatibility endpoints
  • 152.4.8 AWS CloudWatch / Azure Monitor / GCP Cloud Monitoring
  • 152.4.9 PagerDuty Events API v2 compatibility (drop-in)
  • 152.4.10 Opsgenie Integrations API compatibility
  • 152.4.11 Email-to-alert with subject parser
  • 152.4.12 SMS-to-alert for legacy systems
  • 152.4.13 Slack / sovereign chat (Phase 139) command-based alert
  • 152.4.14 Mobile app ad-hoc page
  • 152.4.15 Synthetic monitor (Phase 98) integration

152.5 @neith/notify — Multi-Channel Notification#

  • 152.5.1 Mobile push (sovereign APNS / FCM bridge)
  • 152.5.2 SMS via multi-provider failover (Twilio + sovereign carrier)
  • 152.5.3 Voice call with DTMF ack ("press 4 to acknowledge")
  • 152.5.4 Email (sovereign SMTP from Phase 139)
  • 152.5.5 Slack / Teams / Discord / sovereign chat DM + channel
  • 152.5.6 Web push (PWA)
  • 152.5.7 Custom webhook per user
  • 152.5.8 WhatsApp / Telegram / Signal bot (opt-in)
  • 152.5.9 IRC / Matrix bridge
  • 152.5.10 Channel preference per user + per severity
  • 152.5.11 Do-not-disturb with SEV-1 override
  • 152.5.12 Smart retry across channels
  • 152.5.13 Repeat-page cadence policy
  • 152.5.14 Delivery-receipt tracking
  • 152.5.15 Multi-device reachability

152.6 @neith/noise — Alert Grouping, Noise & ML#

  • 152.6.1 De-duplication by alert key
  • 152.6.2 Time-window grouping into a single incident
  • 152.6.3 Dependency-aware suppression (upstream silences downstream)
  • 152.6.4 Flapping detection with hysteresis
  • 152.6.5 Scheduled maintenance windows (suppress automatically)
  • 152.6.6 ML pattern learner (group-same-cause)
  • 152.6.7 Auto-acknowledge low-confidence noise
  • 152.6.8 Alert fatigue report per team
  • 152.6.9 Symptom vs cause alert ranking
  • 152.6.10 Known-issue linking (auto attach context)

152.7 @neith/war-room — Live Incident War-Room#

  • 152.7.1 Auto-create chat channel on incident (Phase 139)
  • 152.7.2 Auto-create video bridge (Phase 139 / Zoom-class)
  • 152.7.3 Commander / scribe / SME role assignment UI
  • 152.7.4 Live status chips (service / env / customer-impact)
  • 152.7.5 Drag-drop runbook attachment
  • 152.7.6 Action-item capture inline in chat
  • 152.7.7 Decision log timestamped
  • 152.7.8 Live stream of agent actions (Phase 151 EDR response)
  • 152.7.9 Audit recording of voice bridge (consent-gated)
  • 152.7.10 Auto-transcript with speaker tagging
  • 152.7.11 Stakeholder watch-only channel
  • 152.7.12 Role hand-off (new commander) with briefing
  • 152.7.13 Subincident spawn from war-room
  • 152.7.14 Priority bump / down-grade with rationale
  • 152.7.15 Timer widgets (MTTD / MTTR running)

152.8 @neith/status — Public Status Page & Communications#

  • 152.8.1 Customer-facing status page per product
  • 152.8.2 Incident broadcast composer (approve-before-publish)
  • 152.8.3 Component hierarchy + per-component status
  • 152.8.4 Historical uptime percentage (90d rolling)
  • 152.8.5 RSS / Atom / iCal / webhook subscriber feed
  • 152.8.6 Email / SMS subscribers
  • 152.8.7 Custom domain + branding + i18n
  • 152.8.8 Template-library of updates
  • 152.8.9 Audience targeting (by tenant / plan)
  • 152.8.10 Maintenance-window pre-announce
  • 152.8.11 Auto-post from incident-state transitions
  • 152.8.12 Multi-region status (for data-residency offerings)
  • 152.8.13 API for embedding in app
  • 152.8.14 Translations (sovereign translation service)
  • 152.8.15 WCAG AA accessible page

152.9 @neith/runbook — Runbooks, Automation & Playbooks#

  • 152.9.1 Runbook markdown with embedded commands
  • 152.9.2 One-click execute with audit
  • 152.9.3 Parameterised runbook inputs
  • 152.9.4 Suggested-runbook from alert signature
  • 152.9.5 Runbook library per service
  • 152.9.6 Git-backed versioned runbooks
  • 152.9.7 Automated diagnostic collection on trigger
  • 152.9.8 Auto-rollback on Phase 145 CI/CD when alert matches
  • 152.9.9 Auto-scale / failover actions
  • 152.9.10 Conditional playbook branches
  • 152.9.11 Human-in-the-loop approval gates
  • 152.9.12 Dry-run mode
  • 152.9.13 Runbook test fixture
  • 152.9.14 Runbook effectiveness metric
  • 152.9.15 LLM-drafted runbook from past incidents (Phase 47)

152.10 @neith/slo — SLO, Error Budgets & Alerting Policy#

  • 152.10.1 SLI definition (latency / availability / quality / correctness)
  • 152.10.2 SLO target + window definition
  • 152.10.3 Error-budget computation + burn rate
  • 152.10.4 Multi-burn-rate alerts (Google SRE model)
  • 152.10.5 Dependency SLO composition
  • 152.10.6 Business-metric SLO (checkout / activation)
  • 152.10.7 SLO dashboard per service
  • 152.10.8 SLO review cadence
  • 152.10.9 SLO change history
  • 152.10.10 Error-budget policy engine (freeze releases when burned)

152.11 @neith/postmortem — Postmortem, Retro & Learning#

  • 152.11.1 Blameless postmortem template
  • 152.11.2 Auto-populated timeline from war-room events
  • 152.11.3 Auto-populated impact (users, revenue, duration)
  • 152.11.4 Contributing-factors structured fields
  • 152.11.5 Action-item tracker with owner + due-date
  • 152.11.6 Follow-up verification on due-date
  • 152.11.7 Read-review-sign workflow
  • 152.11.8 Publication to internal wiki (Phase 139.10)
  • 152.11.9 Redact-and-share externally (for customers)
  • 152.11.10 Search across past postmortems
  • 152.11.11 Pattern miner (recurring themes)
  • 152.11.12 Knowledge-base linker (Phase 139.9 PKM)
  • 152.11.13 Retro facilitation mode (voting, grouping)
  • 152.11.14 Incident-cost report (SRE toil measured)
  • 152.11.15 LLM summariser producing exec-ready version

152.12 @neith/intel — Metrics, Reporting & SLA#

  • 152.12.1 MTTA / MTTD / MTTR / MTTF per team + service
  • 152.12.2 Page volume + noise + ack-rate metrics
  • 152.12.3 On-call hours per engineer
  • 152.12.4 Customer SLA breach tracking
  • 152.12.5 Post-incident action-item completion rate
  • 152.12.6 Executive digest
  • 152.12.7 DORA-like incident metrics feeding Phase 145
  • 152.12.8 Benchmarks across teams
  • 152.12.9 Drill performance (game-day results)
  • 152.12.10 Export to Phase 142 analytics warehouse

152.13 @neith/customer-incident — Customer Support Bridge#

  • 152.13.1 Zendesk / sovereign support (Phase 139) tickets linked
  • 152.13.2 Auto-attach related tickets to incident
  • 152.13.3 Mass-reply template once resolved
  • 152.13.4 VIP customer flag elevates severity
  • 152.13.5 Impact-segmentation per tenant
  • 152.13.6 CS / AM broadcast to named accounts
  • 152.13.7 Conference-line for enterprise customers
  • 152.13.8 Regulatory-notification workflow (GDPR / HIPAA)
  • 152.13.9 Legal / comms review handoff
  • 152.13.10 Customer-trust center export

152.14 @neith/gameday — Chaos & Game-Day Drills#

  • 152.14.1 Scheduled drill generator (random scenario)
  • 152.14.2 Chaos-engine fault injection triggers
  • 152.14.3 Tabletop exercise facilitator
  • 152.14.4 Scenario library (network partition, region failure, key leak)
  • 152.14.5 Scoring rubric for drill retros
  • 152.14.6 Shadow on-call (trainee mode)
  • 152.14.7 Failure-mode catalogue per service
  • 152.14.8 Auto-generate from postmortem "what if next time"
  • 152.14.9 Recruit / simulate fake customer traffic
  • 152.14.10 Game-day leader-board

152.15 Testing & Validation#

  • 152.15.1 Notification delivery SLA (< 30 s P99)
  • 152.15.2 Failover across SMS / voice providers drill
  • 152.15.3 Schedule fairness audit across teams
  • 152.15.4 Drop-in PagerDuty Events API conformance
  • 152.15.5 Drop-in Opsgenie Integrations API conformance
  • 152.15.6 Status-page cache-bust + high-traffic burst test
  • 152.15.7 War-room start time < 60 s from trigger
  • 152.15.8 Postmortem completeness rubric
  • 152.15.9 Dogfood: manage every Oshun incident on this platform. This is an operational rollout — it needs the live Oshun production incident stream routed through the platform, which has not happened (stated 2026-09-18). blocked:release
  • 152.15.10 External customer pilot (3 teams, 6 months). A six-month field engagement with three external customer teams is a release/adoption event, not an in-repo task (stated 2026-09-18). blocked:release

Implementation and verification evidence: phase-152-evidence.md.