Purpose: Clear guidelines for when and how to escalate incidents during on-call shifts
Quick Reference Escalation Decision Tree#
text
Incident Detected
↓
Assess Severity
↓
┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│ P0 │ │ P1 │ │ P2 │ │ P3 │
└──┬───┘ └──┬───┘ └──┬───┘ └──┬───┘
↓ ↓ ↓ ↓
Immediate 30 mins 2 hours Next Bus Day
Escalation Response Response Response
↓ ↓ ↓ ↓
IC + Platform Platform Team Lead None
Lead Lead
Severity Definitions and Escalation Triggers#
P0 - Critical (Immediate Escalation)#
Criteria#
- Platform Down: >50% of services unavailable
- Data Loss: Confirmed data corruption or loss
- Security Breach: Unauthorized access detected
- Customer Data Exposure: PII/sensitive data leaked
- Revenue Impact: Payment processing completely down
- Core Function Failure: Authentication system down
Escalation Triggers#
- Immediate (0 minutes): Start incident response
- 5 minutes: Engage Incident Commander
- 15 minutes: Escalate to Platform Lead
- 30 minutes: Escalate to Engineering Director
- 60 minutes: Executive notification
Response Team Formation#
text
Primary: On-Call Engineer
Secondary: Incident Commander
Tertiary: Platform Lead
Quaternary: Engineering Director
P1 - High (Timed Escalation)#
Criteria#
- Service Degradation: 20-50% error rates
- Performance Issues: >5s response times
- Partial Outage: Single critical service down with workarounds
- User Impact: >10% of users affected
- Integration Failures: Key third-party service failures
Escalation Triggers#
- 30 minutes: If no significant progress
- 1 hour: Escalate to Platform Lead
- 2 hours: Incident Commander if needed
- 4 hours: Engineering Director notification
Response Team Formation#
text
Primary: On-Call Engineer
Secondary: Platform Lead (after 1 hour)
Tertiary: Subject Matter Expert
P2 - Medium (Business Hours Escalation)#
Criteria#
- Minor Degradation: 5-20% error rates
- Single Component Failure: With functional redundancy
- Non-Critical Issues: Limited user impact
- Monitoring Alerts: Threshold breaches without immediate impact
Escalation Triggers#
- 2 hours: If during business hours
- Next business day: If after hours
- 4 hours: Team Lead notification
Response Team Formation#
text
Primary: On-Call Engineer
Secondary: Team Lead (business hours only)
P3 - Low (No Immediate Escalation)#
Criteria#
- Capacity Warnings: Resource usage alerts
- Monitoring Issues: Dashboard/alerting problems
- Maintenance Notifications: Planned work notifications
- Documentation Issues: Runbook or process gaps
Escalation Triggers#
- Next business day: Normal business process
- Never urgent: Can wait for business hours
Response Team Formation#
text
Primary: On-Call Engineer (acknowledge only)
Secondary: Team Lead (next business day)
Escalation Contact Matrix#
Primary Contacts#
| Role | Primary Contact | Backup Contact | Response SLA |
|---|---|---|---|
| Incident Commander | @incident-commander | @ic-backup | 5 minutes |
| Platform Lead | @platform-lead | @platform-deputy | 15 minutes |
| Security Lead | @security-lead | @security-deputy | 10 minutes |
| Engineering Director | @engineering-director | @eng-director-backup | 30 minutes |
| Database Expert | @dba-lead | @dba-backup | 20 minutes |
| Infrastructure Expert | @infra-lead | @infra-backup | 15 minutes |
| AI/ML Expert | @ai-lead | @ai-backup | 30 minutes |
Contact Methods#
Immediate Response Required (P0)#
- PagerDuty: Page directly
- Phone Call: Use emergency contact numbers
- Slack: Tag in #lilith-incidents with @channel
- SMS: For backup if phone fails
Standard Response (P1)#
- PagerDuty: Send alert
- Slack: Direct message + #lilith-incidents
- Phone: If no response in 30 minutes
Business Hours (P2/P3)#
- Slack: Direct message
- Email: For documentation
- Calendar: Schedule if coordination needed
Escalation Decision Framework#
When to Escalate Immediately#
Technical Criteria#
- Unable to identify root cause within 15 minutes (P0) or 1 hour (P1)
- Multiple systems failing simultaneously
- Requires architecture changes to resolve
- Outside your expertise area (database, security, AI/ML)
- Potential data integrity issues
Business Criteria#
- Customer complaints escalating rapidly
- Media attention or public visibility
- Regulatory implications (security, data protection)
- Revenue impact exceeding thresholds
- SLA breaches imminent or occurring
Operational Criteria#
- Need additional hands for parallel investigation
- Requires coordinated response across multiple teams
- External vendor coordination needed
- Communication strategy required
When NOT to Escalate#
Well-Understood Issues#
- Known problems with documented solutions
- Routine maintenance impacts
- Monitoring false positives you can verify
- Capacity issues with clear scaling path
Minor Issues#
- Single user reports not affecting others
- Cosmetic problems with no functional impact
- Dev/staging environment issues (unless blocking)
- Documentation gaps that don't affect operations
Subject Matter Expert Escalation#
Database Issues#
Escalate when:
- Query performance suddenly degrades
- Connection pool exhaustion
- Replication lag issues
- Data corruption suspected
- Backup/restore needed
Contact: @dba-lead or @dba-backup
Security Incidents#
Escalate when:
- Suspicious access patterns detected
- Authentication bypass attempts
- Data exfiltration suspected
- Vulnerability exploitation
- Compliance violation potential
Contact: @security-lead (immediate for any security concern)
Infrastructure Issues#
Escalate when:
- Kubernetes cluster problems
- Network connectivity issues
- Storage failures
- Load balancer problems
- DNS resolution issues
Contact: @infra-lead or @infra-backup
AI/ML Service Issues#
Escalate when:
- Model inference failures
- GPU resource exhaustion
- Training pipeline failures
- Provider API issues
- Performance degradation >50%
Contact: @ai-lead or @ai-backup
Escalation Communication Templates#
P0 Escalation Message#
text
🚨 P0 INCIDENT - IMMEDIATE ASSISTANCE NEEDED
Incident: [Brief description]
Impact: [User/business impact]
Duration: [How long ongoing]
Investigation: [What you've tried]
Need: [Specific help required]
Channel: #lilith-incidents
P1 Escalation Message#
text
⚠️ P1 INCIDENT - ASSISTANCE NEEDED
Incident: [Brief description]
Severity: P1 - [specific criteria met]
Impact: [Scope of impact]
Investigation: [Current status]
ETA: [If no help, estimated resolution]
Request: [Specific assistance needed]
Subject Matter Expert Request#
text
🔧 SME ASSISTANCE NEEDED
Area: [Database/Security/Infrastructure/AI]
Issue: [Technical problem description]
Context: [What led to this]
Attempted: [What you've tried]
Urgency: [Timeline needs]
Escalation Follow-up Procedures#
After Escalating#
- Document the escalation in incident timeline
- Brief the escalated person on current status
- Continue investigation while waiting for help
- Coordinate actions to avoid conflicting changes
- Update stakeholders on team composition changes
If No Response to Escalation#
- Try backup contact immediately
- Use alternative communication method
- Escalate to next level if still no response
- Document non-response for post-incident review
- Continue with available resources
De-escalation Criteria#
When to Step Down Severity#
- Root cause identified and solution in progress
- Impact reduced significantly (>75% improvement)
- User-facing services restored even if root cause remains
- Workaround implemented that resolves user impact
- Normal operations resumed with monitoring in place
Team Reduction#
- Incident Commander can step down when coordinated response no longer needed
- Subject Matter Experts can step down when their expertise no longer required
- Platform Lead can step down when on-call engineer can handle resolution
- Executive team can step down when business impact eliminated
Escalation Metrics and Review#
Key Metrics#
- Time to escalation: How quickly appropriate help was engaged
- Escalation accuracy: Whether escalation was necessary and to right person
- Response time: How quickly escalated contacts responded
- Resolution improvement: How much escalation helped reduce MTTR
Monthly Review Process#
- Review all escalations from previous month
- Identify patterns in escalation triggers
- Assess contact response times and effectiveness
- Update contact information as needed
- Refine escalation criteria based on learnings
Common Escalation Mistakes#
Over-Escalation#
- Escalating routine issues during learning period
- Calling executives for P2/P3 issues
- Escalating before adequate investigation
- Creating unnecessary urgency
Under-Escalation#
- Trying to solve everything alone
- Not escalating security issues immediately
- Waiting too long on P0/P1 incidents
- Not involving subject matter experts
Poor Communication#
- Vague escalation messages
- Not providing sufficient context
- Failing to brief escalated team members
- Not updating on resolution progress
Remember: Escalation is a tool, not a failure. Use it strategically to ensure the best possible incident response.