# Escalation Matrix

**Purpose**: Clear guidelines for when and how to escalate incidents during
on-call shifts

## Quick Reference Escalation Decision Tree

```
Incident Detected
       ↓
   Assess Severity
       ↓
    ┌──────┐     ┌──────┐     ┌──────┐     ┌──────┐
    │  P0  │     │  P1  │     │  P2  │     │  P3  │
    └──┬───┘     └──┬───┘     └──┬───┘     └──┬───┘
       ↓            ↓            ↓            ↓
  Immediate      30 mins      2 hours    Next Bus Day
  Escalation    Response      Response     Response
       ↓            ↓            ↓            ↓
  IC + Platform   Platform    Team Lead      None
     Lead          Lead
```

## Severity Definitions and Escalation Triggers

### P0 - Critical (Immediate Escalation)

#### Criteria

- **Platform Down**: >50% of services unavailable
- **Data Loss**: Confirmed data corruption or loss
- **Security Breach**: Unauthorized access detected
- **Customer Data Exposure**: PII/sensitive data leaked
- **Revenue Impact**: Payment processing completely down
- **Core Function Failure**: Authentication system down

#### Escalation Triggers

- **Immediate (0 minutes)**: Start incident response
- **5 minutes**: Engage Incident Commander
- **15 minutes**: Escalate to Platform Lead
- **30 minutes**: Escalate to Engineering Director
- **60 minutes**: Executive notification

#### Response Team Formation

```
Primary: On-Call Engineer
Secondary: Incident Commander
Tertiary: Platform Lead
Quaternary: Engineering Director
```

### P1 - High (Timed Escalation)

#### Criteria

- **Service Degradation**: 20-50% error rates
- **Performance Issues**: >5s response times
- **Partial Outage**: Single critical service down with workarounds
- **User Impact**: >10% of users affected
- **Integration Failures**: Key third-party service failures

#### Escalation Triggers

- **30 minutes**: If no significant progress
- **1 hour**: Escalate to Platform Lead
- **2 hours**: Incident Commander if needed
- **4 hours**: Engineering Director notification

#### Response Team Formation

```
Primary: On-Call Engineer
Secondary: Platform Lead (after 1 hour)
Tertiary: Subject Matter Expert
```

### P2 - Medium (Business Hours Escalation)

#### Criteria

- **Minor Degradation**: 5-20% error rates
- **Single Component Failure**: With functional redundancy
- **Non-Critical Issues**: Limited user impact
- **Monitoring Alerts**: Threshold breaches without immediate impact

#### Escalation Triggers

- **2 hours**: If during business hours
- **Next business day**: If after hours
- **4 hours**: Team Lead notification

#### Response Team Formation

```
Primary: On-Call Engineer
Secondary: Team Lead (business hours only)
```

### P3 - Low (No Immediate Escalation)

#### Criteria

- **Capacity Warnings**: Resource usage alerts
- **Monitoring Issues**: Dashboard/alerting problems
- **Maintenance Notifications**: Planned work notifications
- **Documentation Issues**: Runbook or process gaps

#### Escalation Triggers

- **Next business day**: Normal business process
- **Never urgent**: Can wait for business hours

#### Response Team Formation

```
Primary: On-Call Engineer (acknowledge only)
Secondary: Team Lead (next business day)
```

## Escalation Contact Matrix

### Primary Contacts

| Role                      | Primary Contact       | Backup Contact       | Response SLA |
| ------------------------- | --------------------- | -------------------- | ------------ |
| **Incident Commander**    | @incident-commander   | @ic-backup           | 5 minutes    |
| **Platform Lead**         | @platform-lead        | @platform-deputy     | 15 minutes   |
| **Security Lead**         | @security-lead        | @security-deputy     | 10 minutes   |
| **Engineering Director**  | @engineering-director | @eng-director-backup | 30 minutes   |
| **Database Expert**       | @dba-lead             | @dba-backup          | 20 minutes   |
| **Infrastructure Expert** | @infra-lead           | @infra-backup        | 15 minutes   |
| **AI/ML Expert**          | @ai-lead              | @ai-backup           | 30 minutes   |

### Contact Methods

#### Immediate Response Required (P0)

1. **PagerDuty**: Page directly
2. **Phone Call**: Use emergency contact numbers
3. **Slack**: Tag in #lilith-incidents with @channel
4. **SMS**: For backup if phone fails

#### Standard Response (P1)

1. **PagerDuty**: Send alert
2. **Slack**: Direct message + #lilith-incidents
3. **Phone**: If no response in 30 minutes

#### Business Hours (P2/P3)

1. **Slack**: Direct message
2. **Email**: For documentation
3. **Calendar**: Schedule if coordination needed

## Escalation Decision Framework

### When to Escalate Immediately

#### Technical Criteria

- **Unable to identify root cause** within 15 minutes (P0) or 1 hour (P1)
- **Multiple systems failing** simultaneously
- **Requires architecture changes** to resolve
- **Outside your expertise area** (database, security, AI/ML)
- **Potential data integrity issues**

#### Business Criteria

- **Customer complaints** escalating rapidly
- **Media attention** or public visibility
- **Regulatory implications** (security, data protection)
- **Revenue impact** exceeding thresholds
- **SLA breaches** imminent or occurring

#### Operational Criteria

- **Need additional hands** for parallel investigation
- **Requires coordinated response** across multiple teams
- **External vendor coordination** needed
- **Communication strategy** required

### When NOT to Escalate

#### Well-Understood Issues

- **Known problems** with documented solutions
- **Routine maintenance** impacts
- **Monitoring false positives** you can verify
- **Capacity issues** with clear scaling path

#### Minor Issues

- **Single user reports** not affecting others
- **Cosmetic problems** with no functional impact
- **Dev/staging environment** issues (unless blocking)
- **Documentation gaps** that don't affect operations

## Subject Matter Expert Escalation

### Database Issues

**Escalate when:**

- Query performance suddenly degrades
- Connection pool exhaustion
- Replication lag issues
- Data corruption suspected
- Backup/restore needed

**Contact:** @dba-lead or @dba-backup

### Security Incidents

**Escalate when:**

- Suspicious access patterns detected
- Authentication bypass attempts
- Data exfiltration suspected
- Vulnerability exploitation
- Compliance violation potential

**Contact:** @security-lead (immediate for any security concern)

### Infrastructure Issues

**Escalate when:**

- Kubernetes cluster problems
- Network connectivity issues
- Storage failures
- Load balancer problems
- DNS resolution issues

**Contact:** @infra-lead or @infra-backup

### AI/ML Service Issues

**Escalate when:**

- Model inference failures
- GPU resource exhaustion
- Training pipeline failures
- Provider API issues
- Performance degradation >50%

**Contact:** @ai-lead or @ai-backup

## Escalation Communication Templates

### P0 Escalation Message

```
🚨 P0 INCIDENT - IMMEDIATE ASSISTANCE NEEDED

Incident: [Brief description]
Impact: [User/business impact]
Duration: [How long ongoing]
Investigation: [What you've tried]
Need: [Specific help required]
Channel: #lilith-incidents
```

### P1 Escalation Message

```
⚠️ P1 INCIDENT - ASSISTANCE NEEDED

Incident: [Brief description]
Severity: P1 - [specific criteria met]
Impact: [Scope of impact]
Investigation: [Current status]
ETA: [If no help, estimated resolution]
Request: [Specific assistance needed]
```

### Subject Matter Expert Request

```
🔧 SME ASSISTANCE NEEDED

Area: [Database/Security/Infrastructure/AI]
Issue: [Technical problem description]
Context: [What led to this]
Attempted: [What you've tried]
Urgency: [Timeline needs]
```

## Escalation Follow-up Procedures

### After Escalating

1. **Document the escalation** in incident timeline
2. **Brief the escalated person** on current status
3. **Continue investigation** while waiting for help
4. **Coordinate actions** to avoid conflicting changes
5. **Update stakeholders** on team composition changes

### If No Response to Escalation

1. **Try backup contact** immediately
2. **Use alternative communication** method
3. **Escalate to next level** if still no response
4. **Document non-response** for post-incident review
5. **Continue with available resources**

## De-escalation Criteria

### When to Step Down Severity

- **Root cause identified** and solution in progress
- **Impact reduced** significantly (>75% improvement)
- **User-facing services restored** even if root cause remains
- **Workaround implemented** that resolves user impact
- **Normal operations resumed** with monitoring in place

### Team Reduction

- **Incident Commander** can step down when coordinated response no longer
  needed
- **Subject Matter Experts** can step down when their expertise no longer
  required
- **Platform Lead** can step down when on-call engineer can handle resolution
- **Executive team** can step down when business impact eliminated

## Escalation Metrics and Review

### Key Metrics

- **Time to escalation**: How quickly appropriate help was engaged
- **Escalation accuracy**: Whether escalation was necessary and to right person
- **Response time**: How quickly escalated contacts responded
- **Resolution improvement**: How much escalation helped reduce MTTR

### Monthly Review Process

1. **Review all escalations** from previous month
2. **Identify patterns** in escalation triggers
3. **Assess contact response** times and effectiveness
4. **Update contact information** as needed
5. **Refine escalation criteria** based on learnings

## Common Escalation Mistakes

### Over-Escalation

- Escalating routine issues during learning period
- Calling executives for P2/P3 issues
- Escalating before adequate investigation
- Creating unnecessary urgency

### Under-Escalation

- Trying to solve everything alone
- Not escalating security issues immediately
- Waiting too long on P0/P1 incidents
- Not involving subject matter experts

### Poor Communication

- Vague escalation messages
- Not providing sufficient context
- Failing to brief escalated team members
- Not updating on resolution progress

---

**Remember**: Escalation is a tool, not a failure. Use it strategically to
ensure the best possible incident response.
