Incident Response Runbook — Reference

This runbook provides guidance for diagnosing and resolving incidents in the Oshun infrastructure.

Severity Levels#

Level	Definition	Response Time	Examples
SEV-1	Complete outage, all users affected	15 minutes	API down, data loss
SEV-2	Major feature broken, many affected	30 minutes	Payments failing, slow API
SEV-3	Minor feature broken, some affected	2 hours	Single endpoint errors
SEV-4	Cosmetic/minor, minimal impact	Next business	UI glitches, minor bugs

Initial Response#

1. Acknowledge the Incident#

bash

# If using PagerDuty
# Acknowledge via PagerDuty app/web

# Notify in Slack
# Post in #incidents channel:
# "Investigating: [brief description]. ETA for update: 15 mins"

2. Quick Assessment#

Answer these questions:

What is the user-facing impact?
How many users are affected?
When did it start?
Were there any recent deployments?
Are there any ongoing AWS issues? (status.aws.amazon.com)

3. Check Service Health#

bash

# ECS services status
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api oshun-worker oshun-frontend \
  --query 'services[*].{Name:serviceName,Running:runningCount,Desired:desiredCount,Status:status}'

# Recent deployments
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].deployments[*].{Status:status,Created:createdAt,TaskDef:taskDefinition}'

Common Incidents#

API Returning 5xx Errors#

Symptoms#

ALB returning 502/503/504 errors
CloudWatch alarm: High Error Rate
Users reporting "something went wrong"

Diagnosis#

bash

# 1. Check if tasks are running
aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api \
  --desired-status RUNNING

# 2. Check ALB target health
aws elbv2 describe-target-health \
  --target-group-arn $API_TARGET_GROUP_ARN

# 3. Check recent logs for errors
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --filter-pattern "ERROR" \
  --start-time $(date -d '30 minutes ago' +%s000) \
  --limit 50

# 4. Check if there was a recent deployment
aws ecs describe-services \
  --cluster oshun-production \
  --services oshun-api \
  --query 'services[0].events[:10]'

Resolution#

If tasks are unhealthy:

bash

# Force new deployment (restarts all tasks)
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment

If recent deployment is failing:

bash

# Rollback to previous task definition
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --task-definition oshun-api:PREVIOUS_VERSION

If database connection issues:

bash

# Check database connectivity from a task
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "nc -zv $DB_HOST 5432"

High Latency#

Symptoms#

CloudWatch alarm: High Latency
P95 latency > 1s
Users reporting slow responses

Diagnosis#

bash

# 1. Check CPU/memory utilization
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=oshun-production Name=ServiceName,Value=oshun-api \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average Maximum

# 2. Check ALB latency
aws cloudwatch get-metric-statistics \
  --namespace AWS/ApplicationELB \
  --metric-name TargetResponseTime \
  --dimensions Name=LoadBalancer,Value=$ALB_ARN_SUFFIX \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average p95 p99

# 3. Check for slow queries in logs
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --filter-pattern "slow" \
  --start-time $(date -d '1 hour ago' +%s000)

Resolution#

If CPU/memory is high:

bash

# Scale up the service
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --desired-count 5

If database queries are slow:

bash

# Check database performance in RDS console
# or query pg_stat_activity for long-running queries

# Kill problematic queries if needed
psql -h $DB_HOST -U oshun -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE duration > interval '60 seconds';"

Service Not Starting#

Symptoms#

ECS tasks stuck in PENDING
Tasks starting and immediately stopping
CloudWatch alarm: Unhealthy Host Count

Diagnosis#

bash

# 1. Get stopped task details
TASK_ID=$(aws ecs list-tasks \
  --cluster oshun-production \
  --service-name oshun-api \
  --desired-status STOPPED \
  --query 'taskArns[0]' \
  --output text | cut -d'/' -f3)

aws ecs describe-tasks \
  --cluster oshun-production \
  --tasks $TASK_ID \
  --query 'tasks[0].{StopCode:stopCode,StoppedReason:stoppedReason,Containers:containers[*].{Name:name,Reason:reason,ExitCode:exitCode}}'

# 2. Check CloudWatch logs for startup errors
aws logs filter-log-events \
  --log-group-name /ecs/oshun-production/api \
  --start-time $(date -d '15 minutes ago' +%s000) \
  --limit 100

Resolution#

If image pull failure:

bash

# Verify image exists
aws ecr describe-images \
  --repository-name oshun-api \
  --image-ids imageTag=latest

# Check ECR permissions
aws ecr get-repository-policy --repository-name oshun-api

If out of memory:

bash

# Increase task memory (requires new task definition)
# Edit infra/terraform/ecs-services/main.tf
# memory = 2048  # Increase from 1024

# Or manually update via AWS Console
# ECS > Task Definitions > Create new revision > Update memory

If health check failing:

bash

# Test health endpoint locally
curl -v http://localhost:3000/health

# Check if port matches container configuration
aws ecs describe-task-definition \
  --task-definition oshun-api \
  --query 'taskDefinition.containerDefinitions[0].{Port:portMappings[0].containerPort,HealthCheck:healthCheck}'

RunPod Jobs Failing#

Symptoms#

High error rate on RunPod endpoints
Jobs timing out
Empty or corrupted outputs

Diagnosis#

bash

# 1. Check endpoint status via RunPod API
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { id name workersMax workersMin workersDead workersIdle workersRunning } } }"}'

# 2. Check recent job status
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { endpoints { jobs(limit: 10) { id status executionTime error } } } }"}'

# 3. Check worker logs in RunPod dashboard
# Go to: runpod.io > Endpoints > [endpoint] > Logs

Resolution#

If workers are crashing:

Check RunPod dashboard for error logs

Verify Docker image is valid:

bash

docker pull oshunai/runpod-comfyui-sdxl:latest
docker run --rm oshunai/runpod-comfyui-sdxl:latest python -c "print('ok')"

Roll back to previous image version

If queue is backed up:

bash

# Increase max workers temporarily
# Via RunPod dashboard: Endpoints > [endpoint] > Edit > Max Workers

# Or purge queue if jobs are stale
curl -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

Database Connection Issues#

Symptoms#

"connection refused" or "too many connections" errors
Intermittent 500 errors
Slow queries

Diagnosis#

bash

# 1. Check RDS instance status
aws rds describe-db-instances \
  --db-instance-identifier oshun-production \
  --query 'DBInstances[0].{Status:DBInstanceStatus,Connections:DBInstanceIdentifier}'

# 2. Check connection count
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 \
  --statistics Maximum

# 3. Check CPU/memory on RDS
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=oshun-production \
  --start-time $(date -d '1 hour ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 60 \
  --statistics Average Maximum

Resolution#

If too many connections:

bash

# Check for connection leaks in application
# Ensure connection pooling is configured

# Kill idle connections (emergency)
psql -h $DB_HOST -U oshun -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < now() - interval '10 minutes';"

If CPU is high:

bash

# Find expensive queries
psql -h $DB_HOST -U oshun -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 10;"

# Consider upgrading instance class
# Or add read replica for read-heavy workloads

Memory/Disk Full#

Symptoms#

OOM kills in ECS tasks
"No space left on device" errors
Tasks failing to start

Diagnosis#

bash

# 1. Check task memory usage
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "cat /sys/fs/cgroup/memory/memory.usage_in_bytes"

# 2. Check disk usage
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "df -h"

Resolution#

If memory exhausted:

bash

# Increase task memory in task definition
# Or scale horizontally (more tasks, less memory each)

# Restart tasks to clear memory
aws ecs update-service \
  --cluster oshun-production \
  --service oshun-api \
  --force-new-deployment

If disk full:

bash

# Clean up temporary files
aws ecs execute-command \
  --cluster oshun-production \
  --task $TASK_ID \
  --container api \
  --interactive \
  --command "rm -rf /tmp/*"

# For persistent issues, increase ECS task storage
# or mount EFS volume for temporary data

Communication Templates#

Initial Notification#

text

🔴 Incident Declared: [Brief Title]

**Severity:** SEV-[1/2/3]
**Impact:** [Description of user impact]
**Started:** [Time in UTC]
**Lead:** @[your-name]

We are investigating and will provide updates every [15/30] minutes.

Status Update#

text

⏳ Incident Update: [Brief Title]

**Status:** [Investigating/Identified/Monitoring/Resolved]
**Current Actions:**
- [Action 1]
- [Action 2]

**Next Update:** [Time]

Resolution#

text

✅ Incident Resolved: [Brief Title]

**Duration:** [X hours Y minutes]
**Root Cause:** [Brief description]
**Resolution:** [What fixed it]
**Follow-up:** [Ticket for post-mortem/fixes]

Impact summary:
- [Number] users affected
- [Metric] degraded by [X]%

Post-Incident#

1. Create Incident Report#

markdown

# Incident Report: [Title]

**Date:** YYYY-MM-DD **Duration:** X hours Y minutes **Severity:** SEV-X
**Lead:** [Name]

## Summary

[2-3 sentences describing what happened]

## Timeline

- HH:MM UTC - [Event]
- HH:MM UTC - [Event]

## Root Cause

[Description of what caused the incident]

## Resolution

[How the incident was resolved]

## Impact

- Users affected: X
- Requests failed: Y
- Revenue impact: $Z

## Action Items

- [ ] [Action] - Owner - Due Date
- [ ] [Action] - Owner - Due Date

## Lessons Learned

- [Lesson 1]
- [Lesson 2]

2. Schedule Post-Mortem#

SEV-1/2: Within 48 hours
SEV-3: Within 1 week
SEV-4: Optional, as needed

3. Track Action Items#

Create tickets for all action items
Assign owners and due dates
Review in weekly ops meeting

Emergency Contacts#

Role	Contact	Escalation Path
On-Call Engineer	PagerDuty	Primary
Engineering Lead	[Contact]	After 30 min SEV-1
Platform Lead	[Contact]	AWS/Infra issues
CTO	[Contact]	SEV-1 > 1 hour

Useful Links#

CloudWatch Dashboard
ECS Console
RunPod Dashboard
Isis RunPod on-call runbook (planned — not yet written)
Isis creative-pipeline on-call playbook (planned — not yet written)
AWS Status
PagerDuty