Purpose: Structured approach to handle incidents from detection to resolution and post-incident review
Incident Response Overview#
text
Detection → Assessment → Response → Mitigation → Recovery → Review
↓ ↓ ↓ ↓ ↓ ↓
Alert Severity Escalate Fix Issue Verify Learn
Triage Assignment Team Implement Normal Improve
Phase 1: Detection and Initial Response (0-5 minutes)#
1.1 Alert Acknowledgment#
bash
# Acknowledge in PagerDuty
pagerduty ack --incident-id [ID]
# Join incident channel
/join #lilith-incidents
# Start incident timer
!incident start [brief-description]
1.2 Initial Assessment Checklist#
- Alert source and triggering condition
- Affected services and user impact
- Initial symptom verification
- Similar recent incidents
- Current on-call engineer availability
1.3 Severity Assessment (Use Escalation Matrix)#
text
P0: Platform down, data loss, security breach
P1: Service degraded, significant user impact
P2: Minor issues, limited impact
P3: Monitoring alerts, no user impact
1.4 Initial Communication#
text
#lilith-incidents:
🚨 INCIDENT DETECTED [P0/P1/P2/P3]
Service: [affected service]
Symptom: [what we're seeing]
Impact: [user/business impact]
Investigating: @[your-name]
Phase 2: Investigation and Assessment (5-30 minutes)#
2.1 Rapid Health Check#
bash
# Service availability check
./scripts/health-check-all-services.sh
# Infrastructure status
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running
# Recent changes
kubectl rollout history deployment --all-namespaces
git log --oneline --since="1 hour ago"
2.2 Impact Assessment#
text
User Impact:
- Number of affected users: [estimate/exact]
- Affected functionality: [list key features]
- Geographic impact: [regions affected]
- Customer segments: [free/paid/enterprise]
Business Impact:
- Revenue impact: [if applicable]
- SLA breach: [yes/no, which SLAs]
- Customer complaints: [volume/severity]
- External visibility: [public/private]
2.3 Technical Investigation#
bash
# Log analysis for errors
kubectl logs -l app=[service] --since=1h | grep -i error | tail -20
# Resource utilization
kubectl top nodes
kubectl top pods --sort-by=memory --all-namespaces
# Database health
kubectl exec -it postgres-primary-0 -- psql -c "SELECT count(*) FROM pg_stat_activity;"
# External dependencies
curl -f https://status.openai.com/api/v2/status.json
curl -f https://status.aws.amazon.com/data.json
Phase 3: Response Team Formation (If P0/P1)#
3.1 Incident Commander Assignment#
P0 Incidents: Automatic IC assignment P1 Incidents: IC assigned if multiple services affected
3.2 Subject Matter Expert Engagement#
text
Database Issues: @dba-lead
Security Concerns: @security-lead (immediate)
Infrastructure: @infra-lead
AI/ML Services: @ai-lead
External APIs: @integration-lead
3.3 Communication Roles#
- Incident Commander: Overall coordination and decision making
- Technical Lead: Primary investigator and implementer
- Communications Lead: Customer and stakeholder updates
- Scribe: Documentation and timeline tracking
Phase 4: Mitigation and Resolution#
4.1 Immediate Mitigation Options#
Service Scaling#
bash
# Scale up pods
kubectl scale deployment [deployment] --replicas=[count] -n [namespace]
# Add cluster capacity (if needed)
terraform apply -var="node_count=[increased_count]"
Quick Rollback#
bash
# Check recent deployments
kubectl rollout history deployment/[service] -n [namespace]
# Rollback to previous version
kubectl rollout undo deployment/[service] -n [namespace]
# Verify rollback
kubectl rollout status deployment/[service] -n [namespace]
Traffic Management#
bash
# Route traffic away from problematic service
kubectl patch ingress [ingress-name] -p '[traffic-routing-patch]'
# Enable maintenance page
kubectl apply -f maintenance-page.yaml
Database Emergency Actions#
bash
# Kill long-running queries
kubectl exec -it postgres-primary-0 -- psql -c "SELECT pg_terminate_backend([pid]);"
# Increase connection limits temporarily
kubectl exec -it postgres-primary-0 -- psql -c "ALTER SYSTEM SET max_connections = [new_limit];"
4.2 Root Cause Analysis (During Resolution)#
text
Timeline Analysis:
- When did symptoms first appear?
- What changed around that time?
- Pattern correlation with metrics
Change Analysis:
- Recent deployments or configuration changes
- Infrastructure modifications
- Third-party service changes
Data Analysis:
- Error patterns in logs
- Performance metric trends
- Resource utilization patterns
4.3 Solution Implementation#
text
1. Develop hypothesis for root cause
2. Design minimal viable fix
3. Test fix in staging (if possible)
4. Implement fix with monitoring
5. Verify resolution across all affected areas
Phase 5: Verification and Recovery#
5.1 Resolution Verification Checklist#
- All affected services responding normally
- Error rates returned to baseline
- Response times within SLA
- User reports confirming resolution
- Monitoring dashboards showing green
- No new related alerts triggered
5.2 Service Restoration Confirmation#
bash
# End-to-end testing
./scripts/e2e-health-check.sh
# Load testing (if major changes)
./scripts/load-test-verification.sh
# User flow verification
./scripts/critical-path-test.sh
5.3 Communication Updates#
text
Resolution Announcement:
🎉 INCIDENT RESOLVED
Incident: [description]
Duration: [start time] to [end time]
Resolution: [what fixed it]
Impact: [final impact assessment]
Next Steps: [post-incident review timeline]
Phase 6: Post-Incident Activities#
6.1 Immediate Actions (Within 4 hours)#
- Document incident timeline
- Preserve relevant logs and evidence
- Schedule post-incident review
- Update monitoring/alerting if needed
- Brief stakeholders on resolution
6.2 Timeline Documentation Template#
text
Incident Timeline - [Incident ID]
Detection:
[TIME] - Alert triggered by [source]
[TIME] - On-call engineer acknowledged
[TIME] - Initial investigation started
Escalation:
[TIME] - Escalated to [role/person]
[TIME] - [Additional escalations]
Investigation:
[TIME] - [Key findings]
[TIME] - [Hypothesis developed]
[TIME] - [Root cause identified]
Resolution:
[TIME] - [Mitigation actions started]
[TIME] - [Fix implemented]
[TIME] - [Resolution verified]
Communication:
[TIME] - [Customer communications]
[TIME] - [Status page updates]
[TIME] - [Resolution announcement]
Communication Procedures#
Customer-Facing Communications#
Status Page Updates#
text
P0/P1 Incidents: Update within 15 minutes of confirmation
P2 Incidents: Update within 2 hours
P3 Incidents: No customer communication required
Update Template:
"We are currently investigating [issue description].
[Affected services]. Our team is actively working on a resolution.
Updates will be provided every [frequency]."
Support Team Briefing#
text
Incident Brief for Support Team:
Issue: [Non-technical description]
User Impact: [What users will experience]
Workarounds: [If any available]
ETA: [Best current estimate]
Escalation: [When to escalate user contacts]
Internal Communications#
Stakeholder Updates (Every 30 minutes for P0/P1)#
text
Incident Update [#N] - [Timestamp]
Status: [Investigating/Mitigating/Resolved]
Progress: [What we've learned/done]
Current Actions: [What we're doing now]
ETA: [Current estimate]
Next Update: [When next update will be]
Executive Summary (For P0/P1 after 1 hour)#
text
Executive Incident Summary
Incident: [Business-focused description]
Impact: [User and business impact]
Cause: [Root cause if known]
Resolution: [Actions being taken]
Timeline: [Expected resolution]
Follow-up: [Next steps]
Incident Classification Examples#
P0 Examples#
- Authentication service completely down
- Database corruption detected
- Security breach with data access
- Payment processing offline
-
50% of platform unavailable
P1 Examples#
- Single critical service down with no workaround
- Performance degraded to >5s response times
-
20% error rate on core functionality
- Integration failures affecting user workflows
P2 Examples#
- Single non-critical service down
- Performance issues affecting <10% of users
- 5-20% error rates on non-core features
- Third-party service degradation with workarounds
P3 Examples#
- Monitoring alerts without user impact
- Documentation issues
- Dev/staging environment problems
- Capacity warnings without immediate impact
Incident Response Tools#
Required Access#
- PagerDuty (incident management)
- Slack #lilith-incidents (communication)
- Grafana (monitoring dashboards)
- kubectl (Kubernetes access)
- ArgoCD (deployment management)
- Vault (secrets access if needed)
Useful Commands Reference#
bash
# Quick status overview
./scripts/system-health-summary.sh
# Incident timeline capture
./scripts/capture-incident-logs.sh [incident-id]
# Emergency scaling
./scripts/emergency-scale.sh [service] [replicas]
# Traffic diversion
./scripts/traffic-divert.sh [service] [percentage]
# Communication helper
./scripts/incident-notify.sh [severity] [message]
Quality Assurance#
Response Quality Metrics#
- Time to acknowledgment
- Time to correct severity assessment
- Time to appropriate escalation
- Time to user communication
- Time to resolution
Continuous Improvement#
- Monthly incident response training
- Quarterly runbook reviews
- Game day exercises
- Post-incident review action items
Remember: Stay calm, communicate clearly, and focus on user impact. The goal is quick, safe resolution with minimal business impact.