# Incident Response Runbook

This runbook provides guidance for diagnosing and resolving incidents in the
Oshun infrastructure.

## Severity Levels

| Level | Definition                          | Response Time | Examples                   |
| ----- | ----------------------------------- | ------------- | -------------------------- |
| SEV-1 | Complete outage, all users affected | 15 minutes    | API down, data loss        |
| SEV-2 | Major feature broken, many affected | 30 minutes    | Payments failing, slow API |
| SEV-3 | Minor feature broken, some affected | 2 hours       | Single endpoint errors     |
| SEV-4 | Cosmetic/minor, minimal impact      | Next business | UI glitches, minor bugs    |

## Initial Response

### 1. Acknowledge the Incident

```bash
# If using PagerDuty
# Acknowledge via PagerDuty app/web

# Notify in Slack
# Post in #incidents channel:
# "Investigating: [brief description]. ETA for update: 15 mins"
```

### 2. Quick Assessment

Answer these questions:

- [ ] What is the user-facing impact?
- [ ] How many users are affected?
- [ ] When did it start?
- [ ] Were there any recent deployments?
- [ ] Are there any ongoing AWS issues? (status.aws.amazon.com)

### 3. Check Service Health

```bash
# ECS services status
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api oshun-worker oshun-frontend \
  --query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}'

# Recent deployments
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].deployments[*].{Status:status,Created:createdAt,TaskDef:taskDefinition}'
```

## Common Incidents

### API Returning 5xx Errors

#### Symptoms

- ALB returning 502/503/504 errors
- CloudWatch alarm: High Error Rate
- Users reporting "something went wrong"

#### Diagnosis

```bash
# 1. Check if tasks are running
aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api \
  --desired-status RUNNING

# 2. Check ALB target health
aws elbv2 describe-target-health \
  --target-group-arn $API_TARGET_GROUP_ARN

# 3. Check recent logs for errors
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --filter-pattern "ERROR" \
  --start-time $(date -d '30 minutes ago' +%s000) \
  --limit 50

# 4. Check if there was a recent deployment
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].events[:10]'
```

#### Resolution

**If tasks are unhealthy:**

```bash
# Force new deployment (restarts all tasks)
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment
```

**If recent deployment is failing:**

```bash
# Rollback to previous task definition
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --task-definition oshun-api:PREVIOUS_VERSION
```

**If database connection issues:**

```bash
# Check database connectivity from a task
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "nc -zv $DB_HOST 5432"
```

### High Latency

#### Symptoms

- CloudWatch alarm: High Latency
- P95 latency > 1s
- Users reporting slow responses

#### Diagnosis

```bash
# 1. Check CPU/memory utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=oshun-production Name=ServiceName,Value=oshun-api \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average Maximum

# 2. Check ALB latency
aws cloudwatch get-metric-statistics \
  --namespace AWS/ApplicationELB \
  --metric-name TargetResponseTime \
  --dimensions Name=LoadBalancer,Value=$ALB_ARN_SUFFIX \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average p95 p99

# 3. Check for slow queries in logs
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --filter-pattern "slow" \
  --start-time $(date -d '1 hour ago' +%s000)
```

#### Resolution

**If CPU/memory is high:**

```bash
# Scale up the service
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --desired-count 5
```

**If database queries are slow:**

```bash
# Check database performance in RDS console
# or query pg_stat_activity for long-running queries

# Kill problematic queries if needed
psql -h $DB_HOST -U oshun -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE duration > interval '60 seconds';"
```

### Service Not Starting

#### Symptoms

- ECS tasks stuck in PENDING
- Tasks starting and immediately stopping
- CloudWatch alarm: Unhealthy Host Count

#### Diagnosis

```bash
# 1. Get stopped task details
TASK_ID=$(aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api \
  --desired-status STOPPED \
  --query 'taskArns[0]' \
  --output text | cut -d'/' -f3)

aws ecs describe-tasks \
  --cluster oshun-production \
  --tasks $TASK_ID \
  --query 'tasks[0].{StopCode:stopCode,StoppedReason:stoppedReason,Containers:containers[*].{Name:name,Reason:reason,ExitCode:exitCode}}'

# 2. Check CloudWatch logs for startup errors
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --start-time $(date -d '15 minutes ago' +%s000) \
  --limit 100
```

#### Resolution

**If image pull failure:**

```bash
# Verify image exists
aws ecr describe-images \
  --repository-name oshun-api \
  --image-ids imageTag=latest

# Check ECR permissions
aws ecr get-repository-policy --repository-name oshun-api
```

**If out of memory:**

```bash
# Increase task memory (requires new task definition)
# Edit infra/terraform/ecs-services/main.tf
# memory = 2048  # Increase from 1024

# Or manually update via AWS Console
# ECS > Task Definitions > Create new revision > Update memory
```

**If health check failing:**

```bash
# Test health endpoint locally
curl -v http://localhost:3000/health

# Check if port matches container configuration
aws ecs describe-task-definition \
  --task-definition oshun-api \
  --query 'taskDefinition.containerDefinitions[0].{Port:portMappings[0].containerPort,HealthCheck:healthCheck}'
```

### RunPod Jobs Failing

#### Symptoms

- High error rate on RunPod endpoints
- Jobs timing out
- Empty or corrupted outputs

#### Diagnosis

```bash
# 1. Check endpoint status via RunPod API
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { id name workersMax workersMin workersDead workersIdle workersRunning } } }"}'

# 2. Check recent job status
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { jobs(limit: 10) { id status executionTime error } } } }"}'

# 3. Check worker logs in RunPod dashboard
# Go to: runpod.io > Endpoints > [endpoint] > Logs
```

#### Resolution

**If workers are crashing:**

1. Check RunPod dashboard for error logs
2. Verify Docker image is valid:
   ```bash
   docker pull oshunai/runpod-comfyui-sdxl:latest
   docker run --rm oshunai/runpod-comfyui-sdxl:latest python -c "print('ok')"
   ```
3. Roll back to previous image version

**If queue is backed up:**

```bash
# Increase max workers temporarily
# Via RunPod dashboard: Endpoints > [endpoint] > Edit > Max Workers

# Or purge queue if jobs are stale
curl -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"
```

### Database Connection Issues

#### Symptoms

- "connection refused" or "too many connections" errors
- Intermittent 500 errors
- Slow queries

#### Diagnosis

```bash
# 1. Check RDS instance status
aws rds describe-db-instances \
  --db-instance-identifier oshun-production \
  --query 'DBInstances[0].{Status:DBInstanceStatus,Connections:DBInstanceIdentifier}'

# 2. Check connection count
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 \
  --statistics Maximum

# 3. Check CPU/memory on RDS
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 \
  --statistics Average Maximum
```

#### Resolution

**If too many connections:**

```bash
# Check for connection leaks in application
# Ensure connection pooling is configured

# Kill idle connections (emergency)
psql -h $DB_HOST -U oshun -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '10 minutes';"
```

**If CPU is high:**

```bash
# Find expensive queries
psql -h $DB_HOST -U oshun -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 10;"

# Consider upgrading instance class
# Or add read replica for read-heavy workloads
```

### Memory/Disk Full

#### Symptoms

- OOM kills in ECS tasks
- "No space left on device" errors
- Tasks failing to start

#### Diagnosis

```bash
# 1. Check task memory usage
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "cat /sys/fs/cgroup/memory/memory.usage_in_bytes"

# 2. Check disk usage
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "df -h"
```

#### Resolution

**If memory exhausted:**

```bash
# Increase task memory in task definition
# Or scale horizontally (more tasks, less memory each)

# Restart tasks to clear memory
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment
```

**If disk full:**

```bash
# Clean up temporary files
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "rm -rf /tmp/*"

# For persistent issues, increase ECS task storage
# or mount EFS volume for temporary data
```

## Communication Templates

### Initial Notification

```
🔴 Incident Declared: [Brief Title]

**Severity:** SEV-[1/2/3]
**Impact:** [Description of user impact]
**Started:** [Time in UTC]
**Lead:** @[your-name]

We are investigating and will provide updates every [15/30] minutes.
```

### Status Update

```
⏳ Incident Update: [Brief Title]

**Status:** [Investigating/Identified/Monitoring/Resolved]
**Current Actions:**
- [Action 1]
- [Action 2]

**Next Update:** [Time]
```

### Resolution

```
✅ Incident Resolved: [Brief Title]

**Duration:** [X hours Y minutes]
**Root Cause:** [Brief description]
**Resolution:** [What fixed it]
**Follow-up:** [Ticket for post-mortem/fixes]

Impact summary:
- [Number] users affected
- [Metric] degraded by [X]%
```

## Post-Incident

### 1. Create Incident Report

```markdown
# Incident Report: [Title]

**Date:** YYYY-MM-DD **Duration:** X hours Y minutes **Severity:** SEV-X
**Lead:** [Name]

## Summary

[2-3 sentences describing what happened]

## Timeline

- HH:MM UTC - [Event]
- HH:MM UTC - [Event]

## Root Cause

[Description of what caused the incident]

## Resolution

[How the incident was resolved]

## Impact

- Users affected: X
- Requests failed: Y
- Revenue impact: $Z

## Action Items

- [ ] [Action] - Owner - Due Date
- [ ] [Action] - Owner - Due Date

## Lessons Learned

- [Lesson 1]
- [Lesson 2]
```

### 2. Schedule Post-Mortem

- SEV-1/2: Within 48 hours
- SEV-3: Within 1 week
- SEV-4: Optional, as needed

### 3. Track Action Items

- Create tickets for all action items
- Assign owners and due dates
- Review in weekly ops meeting

## Emergency Contacts

| Role             | Contact   | Escalation Path    |
| ---------------- | --------- | ------------------ |
| On-Call Engineer | PagerDuty | Primary            |
| Engineering Lead | [Contact] | After 30 min SEV-1 |
| Platform Lead    | [Contact] | AWS/Infra issues   |
| CTO              | [Contact] | SEV-1 > 1 hour     |

## Useful Links

- [CloudWatch Dashboard](https://console.aws.amazon.com/cloudwatch)
- [ECS Console](https://console.aws.amazon.com/ecs)
- [RunPod Dashboard](https://runpod.io/console/pods)
- Isis RunPod on-call runbook (planned — not yet written)
- Isis creative-pipeline on-call playbook (planned — not yet written)
- [AWS Status](https://status.aws.amazon.com)
- [PagerDuty](https://oshun.pagerduty.com)