Historical task source. The Eve task board holds current task state.
This document defines how Tara responds to production incidents, ensuring consistent and effective handling of service disruptions.
Table of Contents#
- Incident Classification
- Roles and Responsibilities
- Incident Response Process
- Communication Protocols
- Escalation Paths
- On-Call Procedures
- Common Incident Playbooks
- Post-Incident Procedures
- Tools and Resources
Incident Classification#
Severity Levels#
| Severity | Name | Definition | Response Time | Update Frequency |
|---|---|---|---|---|
| P0 | Critical | Complete service outage or data breach | Immediate | Every 15 min |
| P1 | Major | Core feature unavailable, significant impact | 15 minutes | Every 30 min |
| P2 | Moderate | Partial degradation, workaround available | 1 hour | Every 2 hours |
| P3 | Minor | Cosmetic issue, minimal user impact | 4 hours | As needed |
P0 - Critical Incidents#
Examples:
- Complete application unavailability
- Database corruption or data loss
- Security breach or unauthorized access
- Payment system complete failure
- Mass user credential exposure
Response expectations:
- All hands on deck
- Immediate executive notification
- Status page updated immediately
- User communication within 1 hour
P1 - Major Incidents#
Examples:
- Authentication system down
- Meditation playback completely broken
- Subscription purchases failing
- Push notifications system failure
- Significant performance degradation (>500% slower)
Response expectations:
- On-call engineer responds immediately
- Engineering lead notified within 15 minutes
- Status page updated within 15 minutes
- Additional resources pulled if needed
P2 - Moderate Incidents#
Examples:
- Single feature broken (e.g., search not working)
- Slow response times (50-200% slower)
- Intermittent errors (<5% of requests)
- Download feature broken
- Progress sync delayed
Response expectations:
- On-call engineer investigates within 1 hour
- Fix or workaround within 4 hours
- May be handled during business hours
P3 - Minor Incidents#
Examples:
- Visual/UI bugs
- Typos in content
- Non-critical feature edge cases
- Minor performance issues
- Single user reports
Response expectations:
- Tracked for next release
- Addressed during normal development
- No immediate response required
Roles and Responsibilities#
Incident Commander (IC)#
The IC leads the incident response and has final decision authority.
Responsibilities:
- Declare incident severity
- Coordinate response efforts
- Make key decisions
- Manage communication
- Declare incident resolved
Who can be IC:
- On-call engineer (default)
- Engineering lead
- Any senior engineer
Technical Lead#
Leads the technical investigation and resolution.
Responsibilities:
- Diagnose root cause
- Propose and implement fixes
- Coordinate with other engineers
- Document technical findings
Communications Lead#
Manages all internal and external communications.
Responsibilities:
- Update status page
- Send internal notifications
- Draft user communications
- Handle press inquiries (if any)
Scribe#
Documents the incident timeline and actions.
Responsibilities:
- Maintain incident timeline
- Record all decisions
- Track action items
- Prepare postmortem content
Incident Response Process#
Phase 1: Detection (0-5 minutes)#
Alert Triggered / Issue Reported
│
▼
┌─────────────────────────────────┐
│ On-Call Engineer Acknowledges │
│ - Review alert details │
│ - Quick triage │
│ - Initial severity assessment │
└─────────────────────────────────┘
│
▼
Incident Declared
Actions:
- Acknowledge alert in monitoring system
- Quickly assess scope and severity
- Declare incident in #incidents channel
- Assign initial roles (IC = on-call by default)
Phase 2: Triage (5-15 minutes)#
Incident Declared
│
▼
┌─────────────────────────────────┐
│ Gather Information │
│ - Check dashboards │
│ - Review recent deployments │
│ - Check error logs │
│ - Identify affected users │
└─────────────────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Confirm Severity │
│ - P0: Escalate immediately │
│ - P1: Notify engineering lead │
│ - P2/P3: Continue investigation │
└─────────────────────────────────┘
Actions:
- Open monitoring dashboards
- Check recent changes (deployments, config)
- Review error logs and stack traces
- Determine blast radius
- Confirm or adjust severity level
Phase 3: Mitigation (15 minutes - 4 hours)#
Triage Complete
│
▼
┌─────────────────────────────────┐
│ Stabilize the System │
│ - Apply temporary fixes │
│ - Rollback if necessary │
│ - Enable feature flags │
│ - Scale resources │
└─────────────────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Verify Mitigation │
│ - Monitor metrics │
│ - Test affected functionality │
│ - Confirm user reports │
└─────────────────────────────────┘
Actions:
- Implement quickest path to stability
- Prioritize user impact over root cause
- Use established playbooks
- Verify metrics returning to normal
- Update status page with progress
Phase 4: Resolution (Variable)#
System Stabilized
│
▼
┌─────────────────────────────────┐
│ Permanent Fix (if needed) │
│ - Develop proper solution │
│ - Test thoroughly │
│ - Deploy fix │
│ - Verify resolution │
└─────────────────────────────────┘
│
▼
Incident Resolved
Actions:
- Develop permanent fix (if mitigation was temporary)
- Follow normal deployment procedures
- Monitor for recurrence
- Update documentation if needed
Phase 5: Closure#
Incident Resolved
│
▼
┌─────────────────────────────────┐
│ Wrap-Up │
│ - Final status update │
│ - Internal notification │
│ - Schedule postmortem │
│ - Create follow-up tickets │
└─────────────────────────────────┘
Actions:
- Update status page to "Resolved"
- Send final internal communication
- Schedule postmortem (within 48 hours for P0/P1)
- Create tickets for follow-up items
- Thank responders
Communication Protocols#
Internal Communication#
Incident Declaration#
@channel 🚨 INCIDENT DECLARED
Severity: P[0/1/2]
Service: [Affected service]
Impact: [Brief description]
IC: [Name]
Bridge: [Zoom/Meet link if needed]
Updates: Every [15/30] minutes
#incident-[timestamp]
Progress Updates#
📊 INCIDENT UPDATE #[number]
Status: [Investigating / Mitigating / Monitoring]
Progress: [What's been done]
Next: [What's being tried next]
ETA: [If known, otherwise "Unknown"]
IC: [Name]
Resolution#
✅ INCIDENT RESOLVED
Severity: P[0/1/2]
Duration: [Start] - [End] ([total time])
Impact: [Number of users, revenue impact if known]
Resolution: [Brief description]
Postmortem scheduled: [Date/Time]
Thanks to: [Responders]
External Communication#
Status Page Templates#
Investigating:
Title: Investigating Issues with [Service]
We are currently investigating reports of [symptoms].
Some users may experience [impact description].
We will provide updates as we learn more.
Identified:
Title: Issue Identified - [Service]
We have identified the cause of [symptoms].
Our team is working on a fix.
Impact: [Description]
Workaround: [If available]
Next update in [time].
Monitoring:
Title: Fix Deployed - Monitoring [Service]
We have deployed a fix for [issue].
We are monitoring the system to ensure stability.
Most users should no longer experience issues.
We will provide a final update once confirmed resolved.
Resolved:
Title: Resolved - [Service]
The issue affecting [service] has been resolved.
All systems are operating normally.
Duration: [Start] to [End]
Impact: [Brief description]
We apologize for any inconvenience.
Notification Thresholds#
| Event | Internal | Status Page | Users |
|---|---|---|---|
| P0 declared | Immediate | Immediate | Within 30m |
| P1 declared | Immediate | Within 15m | If >1 hour |
| Significant progress | Every 15-30m | Every update | As needed |
| Resolution | Immediate | Immediate | If notified |
| Postmortem scheduled | Within 24h | N/A | N/A |
Escalation Paths#
Engineering Escalation#
On-Call Engineer
│
├── Cannot diagnose → Engineering Lead
│ │
│ ├── Cannot resolve → CTO
│ │
│ └── Needs more resources → Pull team
│
└── Security issue → Security Lead → CTO → Legal
Business Escalation#
P0 Incident Declared
│
├── Immediately notify → Engineering Lead
│ │
├── Within 15 minutes → Product Lead
│ │
└── Within 30 minutes → Executive Team
Escalation Contacts#
| Role | Primary Contact | Backup Contact | Method |
|---|---|---|---|
| On-Call Engineer | PagerDuty | Phone tree | Automated |
| Engineering Lead | [Name] | [Name] | Slack + Phone |
| Product Lead | [Name] | [Name] | Slack + Phone |
| Security Lead | [Name] | [Name] | Slack + Phone |
| CTO | [Name] | [Name] | Phone |
| CEO | [Name] | N/A | Phone |
When to Escalate#
Escalate to Engineering Lead when:
- P0 or P1 incident
- Cannot diagnose within 30 minutes
- Need access not available to on-call
- Decision beyond on-call authority
Escalate to Executive when:
- P0 incident lasting > 1 hour
- Data breach or security incident
- Significant revenue impact
- Potential press/media involvement
On-Call Procedures#
On-Call Schedule#
- Primary: Week-long rotation, Mon 9am to Mon 9am
- Secondary: Backup for primary, shadow role
- Schedule: Maintained in PagerDuty/OpsGenie
On-Call Responsibilities#
-
Respond to alerts within SLA
- P0/P1: 15 minutes (24/7)
- P2: 1 hour (business hours)
-
Investigate and triage
- Determine severity
- Begin initial response
-
Communicate
- Update #incidents channel
- Escalate as needed
-
Handoff
- Document open issues
- Brief next on-call
On-Call Best Practices#
DO:
- Keep your phone charged and with you
- Test alert routing at shift start
- Acknowledge alerts promptly
- Ask for help when stuck
- Document everything
DON'T:
- Let alert fatigue cause missed pages
- Work beyond your capability alone
- Make major changes without review
- Forget to hand off issues
On-Call Handoff Template#
On-Call Handoff: [Date]
Active Issues:
- [Issue 1]: [Status, next steps]
- [Issue 2]: [Status, next steps]
Recent Incidents:
- [Date] P[X]: [Brief description, link to postmortem]
Upcoming Risks:
- [Deployment/change that might cause issues]
Notes for next on-call:
- [Any relevant information]
Common Incident Playbooks#
Playbook: API Unresponsive#
Symptoms:
- Health checks failing
- 5xx errors in monitoring
- Users reporting app errors
Diagnostic Steps:
# 1. Check service health
curl -v https://api.tara.app/health
# 2. Check container/instance status
aws ecs describe-services --cluster tara-production --services tara-api
# 3. Check recent deployments
vercel list tara-api | head -5
# 4. Check logs
aws logs tail /ecs/tara-api --since 30m
# 5. Check database connectivity
aws rds describe-db-instances --db-instance-identifier tara-production
Resolution Options:
- Restart service
- Rollback recent deployment
- Scale up instances
- Fix database connection
- Clear cache
Playbook: Database Connection Issues#
Symptoms:
- Database connection timeouts
- Slow queries
- Connection pool exhausted
Diagnostic Steps:
# 1. Check RDS status
aws rds describe-db-instances --db-instance-identifier tara-production
# 2. Check connection count
# In psql:
SELECT count(*) FROM pg_stat_activity;
# 3. Check for long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;
# 4. Check instance metrics
# CloudWatch: CPUUtilization, FreeableMemory, DatabaseConnections
Resolution Options:
- Kill long-running queries
- Restart connection pool (app restart)
- Scale database instance
- Enable/adjust read replicas
- Optimize problematic queries
Playbook: High Error Rate#
Symptoms:
- Error rate > 1%
- Specific endpoint failing
- Partial functionality broken
Diagnostic Steps:
# 1. Identify affected endpoints
# Check error tracking (Sentry/DataDog)
# 2. Check error logs
aws logs filter-log-events \
--log-group-name /ecs/tara-api \
--filter-pattern "ERROR"
# 3. Check recent changes
git log --oneline -10
# 4. Compare with working version
# Review deployment history
Resolution Options:
- Rollback specific change
- Feature flag disable
- Fix and hotfix
- Scale resources if load-related
Playbook: Performance Degradation#
Symptoms:
- Response times increased
- Users reporting slowness
- Timeouts occurring
Diagnostic Steps:
# 1. Check APM for slow endpoints
# DataDog/New Relic traces
# 2. Check resource utilization
# CPU, Memory, I/O metrics
# 3. Check database performance
# Slow query log, connection stats
# 4. Check cache hit rates
# Redis metrics
# 5. Check CDN/network
# CDN analytics, latency metrics
Resolution Options:
- Scale up/out
- Enable caching
- Optimize queries
- Clear problematic cache
- Enable CDN edge caching
Playbook: Authentication Failures#
Symptoms:
- Users cannot log in
- Token validation failing
- Session errors
Diagnostic Steps:
# 1. Test auth endpoints
curl -X POST https://api.tara.app/auth/login \
-d '{"email":"test@test.com","password":"test"}'
# 2. Check auth service logs
# Look for JWT errors, database errors
# 3. Check third-party auth
# Apple/Google auth status
# 4. Check token secrets
# Verify environment variables
# 5. Check session storage (Redis)
redis-cli ping
Resolution Options:
- Restart auth service
- Verify JWT secrets
- Check third-party provider status
- Clear session cache
- Rollback auth changes
Playbook: Payment Processing Failures#
Symptoms:
- Purchases not completing
- Subscription status incorrect
- Receipt validation failing
Diagnostic Steps:
# 1. Check payment provider status
# RevenueCat dashboard / Stripe dashboard
# 2. Check webhook delivery
# Webhook logs in provider dashboard
# 3. Verify API keys
# Check environment configuration
# 4. Test subscription status
# Query user subscription status
Resolution Options:
- Verify API credentials
- Check webhook endpoints
- Manual subscription verification
- Contact payment provider
- Enable backup payment flow
Post-Incident Procedures#
Postmortem Requirements#
| Severity | Postmortem Required | Timeline |
|---|---|---|
| P0 | Yes | Within 48h |
| P1 | Yes | Within 72h |
| P2 | Recommended | Within 1 week |
| P3 | Optional | If requested |
Postmortem Template#
See Rollback Plan - Postmortem Template
Follow-Up Actions#
After every P0/P1 incident:
- Postmortem meeting - Blameless review with all responders
- Action items - Concrete improvements with owners and deadlines
- Documentation - Update runbooks and playbooks
- Training - Share learnings with broader team
- Metrics - Track MTTR, MTTD improvements
Incident Metrics#
Track these metrics monthly:
| Metric | Definition |
|---|---|
| MTTD (Mean Time to Detect) | Time from issue start to alert |
| MTTA (Mean Time to Acknowledge) | Time from alert to human response |
| MTTR (Mean Time to Resolve) | Time from issue start to resolution |
| Incident count by severity | Number of P0, P1, P2, P3 incidents |
| Alert noise ratio | Actionable alerts / Total alerts |
Tools and Resources#
Monitoring Tools#
| Tool | Purpose | URL |
|---|---|---|
| DataDog | APM, logs, metrics | app.datadoghq.com |
| Sentry | Error tracking | sentry.io |
| PagerDuty | Alerting, on-call | pagerduty.com |
| Status Page | External status | status.tara.app |
Access Requirements#
On-call engineers need:
- AWS Console access
- Vercel access
- Database read access
- Monitoring dashboards
- PagerDuty account
- Status page admin
Quick Links#
| Resource | Link |
|---|---|
| API Dashboard | [Internal link] |
| Database Console | [Internal link] |
| Deployment Log | [Internal link] |
| Runbook Index | [Internal link] |
| On-Call Schedule | [Internal link] |
Emergency Contacts#
| Role | Name | Phone | Availability |
|---|---|---|---|
| On-Call Primary | PagerDuty Primary | PagerDuty | 24/7 |
| Engineering | Tara Platform Lead | PagerDuty + Slack | 24/7 for P0 |
| Security | Security Response Lead | PagerDuty + Slack | 24/7 for P0 |
| Executive | Tara GM | Phone + Email | P0 only |
Last Updated: January 2026
For questions about incident response, contact ops@tara.app