Eve library

Incident Response Plan

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

This document defines how Tara responds to production incidents, ensuring consistent and effective handling of service disruptions.


Table of Contents#

  1. Incident Classification
  2. Roles and Responsibilities
  3. Incident Response Process
  4. Communication Protocols
  5. Escalation Paths
  6. On-Call Procedures
  7. Common Incident Playbooks
  8. Post-Incident Procedures
  9. Tools and Resources

Incident Classification#

Severity Levels#

Severity Name Definition Response Time Update Frequency
P0 Critical Complete service outage or data breach Immediate Every 15 min
P1 Major Core feature unavailable, significant impact 15 minutes Every 30 min
P2 Moderate Partial degradation, workaround available 1 hour Every 2 hours
P3 Minor Cosmetic issue, minimal user impact 4 hours As needed

P0 - Critical Incidents#

Examples:

  • Complete application unavailability
  • Database corruption or data loss
  • Security breach or unauthorized access
  • Payment system complete failure
  • Mass user credential exposure

Response expectations:

  • All hands on deck
  • Immediate executive notification
  • Status page updated immediately
  • User communication within 1 hour

P1 - Major Incidents#

Examples:

  • Authentication system down
  • Meditation playback completely broken
  • Subscription purchases failing
  • Push notifications system failure
  • Significant performance degradation (>500% slower)

Response expectations:

  • On-call engineer responds immediately
  • Engineering lead notified within 15 minutes
  • Status page updated within 15 minutes
  • Additional resources pulled if needed

P2 - Moderate Incidents#

Examples:

  • Single feature broken (e.g., search not working)
  • Slow response times (50-200% slower)
  • Intermittent errors (<5% of requests)
  • Download feature broken
  • Progress sync delayed

Response expectations:

  • On-call engineer investigates within 1 hour
  • Fix or workaround within 4 hours
  • May be handled during business hours

P3 - Minor Incidents#

Examples:

  • Visual/UI bugs
  • Typos in content
  • Non-critical feature edge cases
  • Minor performance issues
  • Single user reports

Response expectations:

  • Tracked for next release
  • Addressed during normal development
  • No immediate response required

Roles and Responsibilities#

Incident Commander (IC)#

The IC leads the incident response and has final decision authority.

Responsibilities:

  • Declare incident severity
  • Coordinate response efforts
  • Make key decisions
  • Manage communication
  • Declare incident resolved

Who can be IC:

  • On-call engineer (default)
  • Engineering lead
  • Any senior engineer

Technical Lead#

Leads the technical investigation and resolution.

Responsibilities:

  • Diagnose root cause
  • Propose and implement fixes
  • Coordinate with other engineers
  • Document technical findings

Communications Lead#

Manages all internal and external communications.

Responsibilities:

  • Update status page
  • Send internal notifications
  • Draft user communications
  • Handle press inquiries (if any)

Scribe#

Documents the incident timeline and actions.

Responsibilities:

  • Maintain incident timeline
  • Record all decisions
  • Track action items
  • Prepare postmortem content

Incident Response Process#

Phase 1: Detection (0-5 minutes)#

text
Alert Triggered / Issue Reported
         │
         ▼
┌─────────────────────────────────┐
│ On-Call Engineer Acknowledges   │
│ - Review alert details          │
│ - Quick triage                  │
│ - Initial severity assessment   │
└─────────────────────────────────┘
         │
         ▼
   Incident Declared

Actions:

  1. Acknowledge alert in monitoring system
  2. Quickly assess scope and severity
  3. Declare incident in #incidents channel
  4. Assign initial roles (IC = on-call by default)

Phase 2: Triage (5-15 minutes)#

text
   Incident Declared
         │
         ▼
┌─────────────────────────────────┐
│ Gather Information              │
│ - Check dashboards              │
│ - Review recent deployments     │
│ - Check error logs              │
│ - Identify affected users       │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│ Confirm Severity                │
│ - P0: Escalate immediately      │
│ - P1: Notify engineering lead   │
│ - P2/P3: Continue investigation │
└─────────────────────────────────┘

Actions:

  1. Open monitoring dashboards
  2. Check recent changes (deployments, config)
  3. Review error logs and stack traces
  4. Determine blast radius
  5. Confirm or adjust severity level

Phase 3: Mitigation (15 minutes - 4 hours)#

text
  Triage Complete
         │
         ▼
┌─────────────────────────────────┐
│ Stabilize the System            │
│ - Apply temporary fixes         │
│ - Rollback if necessary         │
│ - Enable feature flags          │
│ - Scale resources               │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│ Verify Mitigation               │
│ - Monitor metrics               │
│ - Test affected functionality   │
│ - Confirm user reports          │
└─────────────────────────────────┘

Actions:

  1. Implement quickest path to stability
  2. Prioritize user impact over root cause
  3. Use established playbooks
  4. Verify metrics returning to normal
  5. Update status page with progress

Phase 4: Resolution (Variable)#

text
 System Stabilized
         │
         ▼
┌─────────────────────────────────┐
│ Permanent Fix (if needed)       │
│ - Develop proper solution       │
│ - Test thoroughly               │
│ - Deploy fix                    │
│ - Verify resolution             │
└─────────────────────────────────┘
         │
         ▼
 Incident Resolved

Actions:

  1. Develop permanent fix (if mitigation was temporary)
  2. Follow normal deployment procedures
  3. Monitor for recurrence
  4. Update documentation if needed

Phase 5: Closure#

text
 Incident Resolved
         │
         ▼
┌─────────────────────────────────┐
│ Wrap-Up                         │
│ - Final status update           │
│ - Internal notification         │
│ - Schedule postmortem           │
│ - Create follow-up tickets      │
└─────────────────────────────────┘

Actions:

  1. Update status page to "Resolved"
  2. Send final internal communication
  3. Schedule postmortem (within 48 hours for P0/P1)
  4. Create tickets for follow-up items
  5. Thank responders

Communication Protocols#

Internal Communication#

Incident Declaration#

text
@channel 🚨 INCIDENT DECLARED

Severity: P[0/1/2]
Service: [Affected service]
Impact: [Brief description]
IC: [Name]

Bridge: [Zoom/Meet link if needed]
Updates: Every [15/30] minutes

#incident-[timestamp]

Progress Updates#

text
📊 INCIDENT UPDATE #[number]

Status: [Investigating / Mitigating / Monitoring]
Progress: [What's been done]
Next: [What's being tried next]
ETA: [If known, otherwise "Unknown"]

IC: [Name]

Resolution#

text
✅ INCIDENT RESOLVED

Severity: P[0/1/2]
Duration: [Start] - [End] ([total time])
Impact: [Number of users, revenue impact if known]
Resolution: [Brief description]

Postmortem scheduled: [Date/Time]

Thanks to: [Responders]

External Communication#

Status Page Templates#

Investigating:

text
Title: Investigating Issues with [Service]

We are currently investigating reports of [symptoms].
Some users may experience [impact description].

We will provide updates as we learn more.

Identified:

text
Title: Issue Identified - [Service]

We have identified the cause of [symptoms].
Our team is working on a fix.

Impact: [Description]
Workaround: [If available]

Next update in [time].

Monitoring:

text
Title: Fix Deployed - Monitoring [Service]

We have deployed a fix for [issue].
We are monitoring the system to ensure stability.

Most users should no longer experience issues.

We will provide a final update once confirmed resolved.

Resolved:

text
Title: Resolved - [Service]

The issue affecting [service] has been resolved.
All systems are operating normally.

Duration: [Start] to [End]
Impact: [Brief description]

We apologize for any inconvenience.

Notification Thresholds#

Event Internal Status Page Users
P0 declared Immediate Immediate Within 30m
P1 declared Immediate Within 15m If >1 hour
Significant progress Every 15-30m Every update As needed
Resolution Immediate Immediate If notified
Postmortem scheduled Within 24h N/A N/A

Escalation Paths#

Engineering Escalation#

text
On-Call Engineer
       │
       ├── Cannot diagnose → Engineering Lead
       │                            │
       │                            ├── Cannot resolve → CTO
       │                            │
       │                            └── Needs more resources → Pull team
       │
       └── Security issue → Security Lead → CTO → Legal

Business Escalation#

text
P0 Incident Declared
       │
       ├── Immediately notify → Engineering Lead
       │                              │
       ├── Within 15 minutes → Product Lead
       │                              │
       └── Within 30 minutes → Executive Team

Escalation Contacts#

Role Primary Contact Backup Contact Method
On-Call Engineer PagerDuty Phone tree Automated
Engineering Lead [Name] [Name] Slack + Phone
Product Lead [Name] [Name] Slack + Phone
Security Lead [Name] [Name] Slack + Phone
CTO [Name] [Name] Phone
CEO [Name] N/A Phone

When to Escalate#

Escalate to Engineering Lead when:

  • P0 or P1 incident
  • Cannot diagnose within 30 minutes
  • Need access not available to on-call
  • Decision beyond on-call authority

Escalate to Executive when:

  • P0 incident lasting > 1 hour
  • Data breach or security incident
  • Significant revenue impact
  • Potential press/media involvement

On-Call Procedures#

On-Call Schedule#

  • Primary: Week-long rotation, Mon 9am to Mon 9am
  • Secondary: Backup for primary, shadow role
  • Schedule: Maintained in PagerDuty/OpsGenie

On-Call Responsibilities#

  1. Respond to alerts within SLA

    • P0/P1: 15 minutes (24/7)
    • P2: 1 hour (business hours)
  2. Investigate and triage

    • Determine severity
    • Begin initial response
  3. Communicate

    • Update #incidents channel
    • Escalate as needed
  4. Handoff

    • Document open issues
    • Brief next on-call

On-Call Best Practices#

DO:

  • Keep your phone charged and with you
  • Test alert routing at shift start
  • Acknowledge alerts promptly
  • Ask for help when stuck
  • Document everything

DON'T:

  • Let alert fatigue cause missed pages
  • Work beyond your capability alone
  • Make major changes without review
  • Forget to hand off issues

On-Call Handoff Template#

text
On-Call Handoff: [Date]

Active Issues:
- [Issue 1]: [Status, next steps]
- [Issue 2]: [Status, next steps]

Recent Incidents:
- [Date] P[X]: [Brief description, link to postmortem]

Upcoming Risks:
- [Deployment/change that might cause issues]

Notes for next on-call:
- [Any relevant information]

Common Incident Playbooks#

Playbook: API Unresponsive#

Symptoms:

  • Health checks failing
  • 5xx errors in monitoring
  • Users reporting app errors

Diagnostic Steps:

bash
# 1. Check service health
curl -v https://api.tara.app/health

# 2. Check container/instance status
aws ecs describe-services --cluster tara-production --services tara-api

# 3. Check recent deployments
vercel list tara-api | head -5

# 4. Check logs
aws logs tail /ecs/tara-api --since 30m

# 5. Check database connectivity
aws rds describe-db-instances --db-instance-identifier tara-production

Resolution Options:

  1. Restart service
  2. Rollback recent deployment
  3. Scale up instances
  4. Fix database connection
  5. Clear cache

Playbook: Database Connection Issues#

Symptoms:

  • Database connection timeouts
  • Slow queries
  • Connection pool exhausted

Diagnostic Steps:

bash
# 1. Check RDS status
aws rds describe-db-instances --db-instance-identifier tara-production

# 2. Check connection count
# In psql:
SELECT count(*) FROM pg_stat_activity;

# 3. Check for long-running queries
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state = 'active'
ORDER BY duration DESC;

# 4. Check instance metrics
# CloudWatch: CPUUtilization, FreeableMemory, DatabaseConnections

Resolution Options:

  1. Kill long-running queries
  2. Restart connection pool (app restart)
  3. Scale database instance
  4. Enable/adjust read replicas
  5. Optimize problematic queries

Playbook: High Error Rate#

Symptoms:

  • Error rate > 1%
  • Specific endpoint failing
  • Partial functionality broken

Diagnostic Steps:

bash
# 1. Identify affected endpoints
# Check error tracking (Sentry/DataDog)

# 2. Check error logs
aws logs filter-log-events \
  --log-group-name /ecs/tara-api \
  --filter-pattern "ERROR"

# 3. Check recent changes
git log --oneline -10

# 4. Compare with working version
# Review deployment history

Resolution Options:

  1. Rollback specific change
  2. Feature flag disable
  3. Fix and hotfix
  4. Scale resources if load-related

Playbook: Performance Degradation#

Symptoms:

  • Response times increased
  • Users reporting slowness
  • Timeouts occurring

Diagnostic Steps:

bash
# 1. Check APM for slow endpoints
# DataDog/New Relic traces

# 2. Check resource utilization
# CPU, Memory, I/O metrics

# 3. Check database performance
# Slow query log, connection stats

# 4. Check cache hit rates
# Redis metrics

# 5. Check CDN/network
# CDN analytics, latency metrics

Resolution Options:

  1. Scale up/out
  2. Enable caching
  3. Optimize queries
  4. Clear problematic cache
  5. Enable CDN edge caching

Playbook: Authentication Failures#

Symptoms:

  • Users cannot log in
  • Token validation failing
  • Session errors

Diagnostic Steps:

bash
# 1. Test auth endpoints
curl -X POST https://api.tara.app/auth/login \
  -d '{"email":"test@test.com","password":"test"}'

# 2. Check auth service logs
# Look for JWT errors, database errors

# 3. Check third-party auth
# Apple/Google auth status

# 4. Check token secrets
# Verify environment variables

# 5. Check session storage (Redis)
redis-cli ping

Resolution Options:

  1. Restart auth service
  2. Verify JWT secrets
  3. Check third-party provider status
  4. Clear session cache
  5. Rollback auth changes

Playbook: Payment Processing Failures#

Symptoms:

  • Purchases not completing
  • Subscription status incorrect
  • Receipt validation failing

Diagnostic Steps:

bash
# 1. Check payment provider status
# RevenueCat dashboard / Stripe dashboard

# 2. Check webhook delivery
# Webhook logs in provider dashboard

# 3. Verify API keys
# Check environment configuration

# 4. Test subscription status
# Query user subscription status

Resolution Options:

  1. Verify API credentials
  2. Check webhook endpoints
  3. Manual subscription verification
  4. Contact payment provider
  5. Enable backup payment flow

Post-Incident Procedures#

Postmortem Requirements#

Severity Postmortem Required Timeline
P0 Yes Within 48h
P1 Yes Within 72h
P2 Recommended Within 1 week
P3 Optional If requested

Postmortem Template#

See Rollback Plan - Postmortem Template

Follow-Up Actions#

After every P0/P1 incident:

  1. Postmortem meeting - Blameless review with all responders
  2. Action items - Concrete improvements with owners and deadlines
  3. Documentation - Update runbooks and playbooks
  4. Training - Share learnings with broader team
  5. Metrics - Track MTTR, MTTD improvements

Incident Metrics#

Track these metrics monthly:

Metric Definition
MTTD (Mean Time to Detect) Time from issue start to alert
MTTA (Mean Time to Acknowledge) Time from alert to human response
MTTR (Mean Time to Resolve) Time from issue start to resolution
Incident count by severity Number of P0, P1, P2, P3 incidents
Alert noise ratio Actionable alerts / Total alerts

Tools and Resources#

Monitoring Tools#

Tool Purpose URL
DataDog APM, logs, metrics app.datadoghq.com
Sentry Error tracking sentry.io
PagerDuty Alerting, on-call pagerduty.com
Status Page External status status.tara.app

Access Requirements#

On-call engineers need:

  • AWS Console access
  • Vercel access
  • Database read access
  • Monitoring dashboards
  • PagerDuty account
  • Status page admin
Resource Link
API Dashboard [Internal link]
Database Console [Internal link]
Deployment Log [Internal link]
Runbook Index [Internal link]
On-Call Schedule [Internal link]

Emergency Contacts#

Role Name Phone Availability
On-Call Primary PagerDuty Primary PagerDuty 24/7
Engineering Tara Platform Lead PagerDuty + Slack 24/7 for P0
Security Security Response Lead PagerDuty + Slack 24/7 for P0
Executive Tara GM Phone + Email P0 only

Last Updated: January 2026

For questions about incident response, contact ops@tara.app