Historical task source. The Eve task board holds current task state.
Phase 152: Sovereign Incident Management & On-Call Platform — PagerDuty / Opsgenie / VictorOps (Splunk On-Call) / xMatters / FireHydrant / incident.io / Rootly / Blameless / Grafana OnCall / Better Stack Replacement#
Closes the incident-management sovereignty gap. Phase 137 briefly mentions "PagerDuty / sovereign" for live-ops. Phase 149 (Crash Analytics) and Phase 151 (EDR) both fire alerts but neither manages schedules, escalations, SLAs, stakeholder comms, post-mortems, or retros. This phase delivers an end-to-end incident platform for Oshun and external customers.
Sovereign replacement for PagerDuty, Opsgenie, Splunk On-Call (VictorOps), xMatters, FireHydrant, incident.io, Rootly, Blameless, Jeli, Grafana OnCall, Better Stack Uptime / Incidents, statuspage.io, StatusGator, ilert, and Squadcast.
152.1 @neith/incident-core — Incident Model & Lifecycle#
- 152.1.1 Incident entity (id / title / severity / services / owner)
- 152.1.2 Severity scale SEV-1 → SEV-5 with auto-bump rules
- 152.1.3 Incident lifecycle (triggered / investigating / identified / mitigating / resolved)
- 152.1.4 Status transitions with required fields
- 152.1.5 Services / components taxonomy
- 152.1.6 Dependency graph (upstream / downstream blast-radius)
- 152.1.7 Incident linking (parent / child / related)
- 152.1.8 Incident merge + split with full audit
- 152.1.9 Custom fields per-team
- 152.1.10 Tags + saved filters
- 152.1.11 Timeline with automatic + manual entries
- 152.1.12 Impact estimation (users / revenue / SLO burn)
- 152.1.13 Incident closeout checklist
- 152.1.14 Postmortem doc auto-scaffold on resolve
- 152.1.15 Retention + anonymisation for long-term storage
152.2 @neith/oncall — On-Call Schedules & Rotations#
- 152.2.1 Weekly / daily / follow-the-sun rotation templates
- 152.2.2 Layered schedule (primary / secondary / manager)
- 152.2.3 Override window with audit trail
- 152.2.4 Holiday + PTO calendar import (iCal / Google / sovereign cal)
- 152.2.5 Shift swap request + approval
- 152.2.6 On-call time-budget report (hours per person per period)
- 152.2.7 Fair-distribution scheduler suggestions
- 152.2.8 Regional / timezone aware routing
- 152.2.9 Team rotation + escalation matrix composer
- 152.2.10 Current + upcoming on-call view (who to page now)
- 152.2.11 "Who is on-call" API for other tools
- 152.2.12 Cross-team pager hand-off
- 152.2.13 Burnout detection heuristic (nights / weekends / page count)
- 152.2.14 Vacation embargo windows
- 152.2.15 Trainee / shadow mode (dual-page with senior)
152.3 @neith/escalation — Escalation Policies & Routing#
- 152.3.1 Policy DSL (step → notify → wait → escalate)
- 152.3.2 Round-robin + all-at-once + parallel modes
- 152.3.3 Ack timer per step (configurable)
- 152.3.4 Escalate on no-ack → next level
- 152.3.5 Escalate on reject / unavailable
- 152.3.6 Conditional routing (service / env / time-of-day / severity)
- 152.3.7 Team-based routing (CODEOWNERS / service catalog)
- 152.3.8 Role-based routing (incident commander, scribe, SME)
- 152.3.9 Fallback to manager tree
- 152.3.10 Loop prevention / max-cycle guard
152.4 @neith/alert-ingest — Alert Ingestion & Routing#
- 152.4.1 Generic webhook endpoint with signature verify
- 152.4.2 Native integration: Phase 149 (crash)
- 152.4.3 Native integration: Phase 151 (EDR)
- 152.4.4 Native integration: Phase 98 (Nous observability)
- 152.4.5 Prometheus Alertmanager sink
- 152.4.6 Grafana alert sink
- 152.4.7 Datadog / New Relic / Splunk compatibility endpoints
- 152.4.8 AWS CloudWatch / Azure Monitor / GCP Cloud Monitoring
- 152.4.9 PagerDuty Events API v2 compatibility (drop-in)
- 152.4.10 Opsgenie Integrations API compatibility
- 152.4.11 Email-to-alert with subject parser
- 152.4.12 SMS-to-alert for legacy systems
- 152.4.13 Slack / sovereign chat (Phase 139) command-based alert
- 152.4.14 Mobile app ad-hoc page
- 152.4.15 Synthetic monitor (Phase 98) integration
152.5 @neith/notify — Multi-Channel Notification#
- 152.5.1 Mobile push (sovereign APNS / FCM bridge)
- 152.5.2 SMS via multi-provider failover (Twilio + sovereign carrier)
- 152.5.3 Voice call with DTMF ack ("press 4 to acknowledge")
- 152.5.4 Email (sovereign SMTP from Phase 139)
- 152.5.5 Slack / Teams / Discord / sovereign chat DM + channel
- 152.5.6 Web push (PWA)
- 152.5.7 Custom webhook per user
- 152.5.8 WhatsApp / Telegram / Signal bot (opt-in)
- 152.5.9 IRC / Matrix bridge
- 152.5.10 Channel preference per user + per severity
- 152.5.11 Do-not-disturb with SEV-1 override
- 152.5.12 Smart retry across channels
- 152.5.13 Repeat-page cadence policy
- 152.5.14 Delivery-receipt tracking
- 152.5.15 Multi-device reachability
152.6 @neith/noise — Alert Grouping, Noise & ML#
- 152.6.1 De-duplication by alert key
- 152.6.2 Time-window grouping into a single incident
- 152.6.3 Dependency-aware suppression (upstream silences downstream)
- 152.6.4 Flapping detection with hysteresis
- 152.6.5 Scheduled maintenance windows (suppress automatically)
- 152.6.6 ML pattern learner (group-same-cause)
- 152.6.7 Auto-acknowledge low-confidence noise
- 152.6.8 Alert fatigue report per team
- 152.6.9 Symptom vs cause alert ranking
- 152.6.10 Known-issue linking (auto attach context)
152.7 @neith/war-room — Live Incident War-Room#
- 152.7.1 Auto-create chat channel on incident (Phase 139)
- 152.7.2 Auto-create video bridge (Phase 139 / Zoom-class)
- 152.7.3 Commander / scribe / SME role assignment UI
- 152.7.4 Live status chips (service / env / customer-impact)
- 152.7.5 Drag-drop runbook attachment
- 152.7.6 Action-item capture inline in chat
- 152.7.7 Decision log timestamped
- 152.7.8 Live stream of agent actions (Phase 151 EDR response)
- 152.7.9 Audit recording of voice bridge (consent-gated)
- 152.7.10 Auto-transcript with speaker tagging
- 152.7.11 Stakeholder watch-only channel
- 152.7.12 Role hand-off (new commander) with briefing
- 152.7.13 Subincident spawn from war-room
- 152.7.14 Priority bump / down-grade with rationale
- 152.7.15 Timer widgets (MTTD / MTTR running)
152.8 @neith/status — Public Status Page & Communications#
- 152.8.1 Customer-facing status page per product
- 152.8.2 Incident broadcast composer (approve-before-publish)
- 152.8.3 Component hierarchy + per-component status
- 152.8.4 Historical uptime percentage (90d rolling)
- 152.8.5 RSS / Atom / iCal / webhook subscriber feed
- 152.8.6 Email / SMS subscribers
- 152.8.7 Custom domain + branding + i18n
- 152.8.8 Template-library of updates
- 152.8.9 Audience targeting (by tenant / plan)
- 152.8.10 Maintenance-window pre-announce
- 152.8.11 Auto-post from incident-state transitions
- 152.8.12 Multi-region status (for data-residency offerings)
- 152.8.13 API for embedding in app
- 152.8.14 Translations (sovereign translation service)
- 152.8.15 WCAG AA accessible page
152.9 @neith/runbook — Runbooks, Automation & Playbooks#
- 152.9.1 Runbook markdown with embedded commands
- 152.9.2 One-click execute with audit
- 152.9.3 Parameterised runbook inputs
- 152.9.4 Suggested-runbook from alert signature
- 152.9.5 Runbook library per service
- 152.9.6 Git-backed versioned runbooks
- 152.9.7 Automated diagnostic collection on trigger
- 152.9.8 Auto-rollback on Phase 145 CI/CD when alert matches
- 152.9.9 Auto-scale / failover actions
- 152.9.10 Conditional playbook branches
- 152.9.11 Human-in-the-loop approval gates
- 152.9.12 Dry-run mode
- 152.9.13 Runbook test fixture
- 152.9.14 Runbook effectiveness metric
- 152.9.15 LLM-drafted runbook from past incidents (Phase 47)
152.10 @neith/slo — SLO, Error Budgets & Alerting Policy#
- 152.10.1 SLI definition (latency / availability / quality / correctness)
- 152.10.2 SLO target + window definition
- 152.10.3 Error-budget computation + burn rate
- 152.10.4 Multi-burn-rate alerts (Google SRE model)
- 152.10.5 Dependency SLO composition
- 152.10.6 Business-metric SLO (checkout / activation)
- 152.10.7 SLO dashboard per service
- 152.10.8 SLO review cadence
- 152.10.9 SLO change history
- 152.10.10 Error-budget policy engine (freeze releases when burned)
152.11 @neith/postmortem — Postmortem, Retro & Learning#
- 152.11.1 Blameless postmortem template
- 152.11.2 Auto-populated timeline from war-room events
- 152.11.3 Auto-populated impact (users, revenue, duration)
- 152.11.4 Contributing-factors structured fields
- 152.11.5 Action-item tracker with owner + due-date
- 152.11.6 Follow-up verification on due-date
- 152.11.7 Read-review-sign workflow
- 152.11.8 Publication to internal wiki (Phase 139.10)
- 152.11.9 Redact-and-share externally (for customers)
- 152.11.10 Search across past postmortems
- 152.11.11 Pattern miner (recurring themes)
- 152.11.12 Knowledge-base linker (Phase 139.9 PKM)
- 152.11.13 Retro facilitation mode (voting, grouping)
- 152.11.14 Incident-cost report (SRE toil measured)
- 152.11.15 LLM summariser producing exec-ready version
152.12 @neith/intel — Metrics, Reporting & SLA#
- 152.12.1 MTTA / MTTD / MTTR / MTTF per team + service
- 152.12.2 Page volume + noise + ack-rate metrics
- 152.12.3 On-call hours per engineer
- 152.12.4 Customer SLA breach tracking
- 152.12.5 Post-incident action-item completion rate
- 152.12.6 Executive digest
- 152.12.7 DORA-like incident metrics feeding Phase 145
- 152.12.8 Benchmarks across teams
- 152.12.9 Drill performance (game-day results)
- 152.12.10 Export to Phase 142 analytics warehouse
152.13 @neith/customer-incident — Customer Support Bridge#
- 152.13.1 Zendesk / sovereign support (Phase 139) tickets linked
- 152.13.2 Auto-attach related tickets to incident
- 152.13.3 Mass-reply template once resolved
- 152.13.4 VIP customer flag elevates severity
- 152.13.5 Impact-segmentation per tenant
- 152.13.6 CS / AM broadcast to named accounts
- 152.13.7 Conference-line for enterprise customers
- 152.13.8 Regulatory-notification workflow (GDPR / HIPAA)
- 152.13.9 Legal / comms review handoff
- 152.13.10 Customer-trust center export
152.14 @neith/gameday — Chaos & Game-Day Drills#
- 152.14.1 Scheduled drill generator (random scenario)
- 152.14.2 Chaos-engine fault injection triggers
- 152.14.3 Tabletop exercise facilitator
- 152.14.4 Scenario library (network partition, region failure, key leak)
- 152.14.5 Scoring rubric for drill retros
- 152.14.6 Shadow on-call (trainee mode)
- 152.14.7 Failure-mode catalogue per service
- 152.14.8 Auto-generate from postmortem "what if next time"
- 152.14.9 Recruit / simulate fake customer traffic
- 152.14.10 Game-day leader-board
152.15 Testing & Validation#
- 152.15.1 Notification delivery SLA (< 30 s P99)
- 152.15.2 Failover across SMS / voice providers drill
- 152.15.3 Schedule fairness audit across teams
- 152.15.4 Drop-in PagerDuty Events API conformance
- 152.15.5 Drop-in Opsgenie Integrations API conformance
- 152.15.6 Status-page cache-bust + high-traffic burst test
- 152.15.7 War-room start time < 60 s from trigger
- 152.15.8 Postmortem completeness rubric
- 152.15.9 Dogfood: manage every Oshun incident on this platform. This
is an operational rollout — it needs the live Oshun production incident
stream routed through the platform, which has not happened (stated
2026-09-18).
blocked:release - 152.15.10 External customer pilot (3 teams, 6 months). A six-month
field engagement with three external customer teams is a release/adoption
event, not an in-repo task (stated 2026-09-18).
blocked:release
Implementation and verification evidence: phase-152-evidence.md.