Eve library

Disaster Recovery Playbooks

Runbook

Revision 1

On this page

Purpose: Business continuity procedures for catastrophic failures affecting multiple systems or entire regions

DR Overview and Philosophy#

DR Objectives#

  1. Minimize Downtime - Restore critical services as quickly as possible
  2. Preserve Data Integrity - Ensure no data loss during recovery
  3. Maintain Security - Follow security protocols even during emergency recovery
  4. Document Actions - Track all recovery actions for post-event analysis
  5. Communicate Status - Keep stakeholders informed throughout recovery

DR Scenarios Covered#

  • Database Failures - Primary database corruption or complete failure
  • Service-Level Disasters - Multiple microservices failing simultaneously
  • Network/Infrastructure Failures - Regional network outages or cluster failures
  • Security Incidents - Compromise requiring full system isolation and rebuild
  • Data Center Outages - Complete loss of primary hosting region

DR Activation Criteria#

Automatic DR Triggers#

  • Primary database cluster completely unavailable for >30 minutes
  • >75% of core services failing simultaneously
  • Complete regional failure of primary hosting environment
  • Security compromise requiring full system isolation

Manual DR Decision Points#

  • Estimated recovery time for normal repair >4 hours
  • Data corruption detected requiring restore from backup
  • Cascading failures indicating systemic infrastructure issues
  • Business continuity requirements mandating failover

DR Decision Matrix#

text
Impact Level    |  Affected Services  |  Estimated Recovery  |  DR Action
Critical (P0)   |  >75% services      |  >4 hours           |  Full DR
High (P1)       |  50-75% services    |  >6 hours           |  Partial DR
Medium (P2)     |  25-50% services    |  >8 hours           |  Consider DR
Low (P3)        |  <25% services      |  Any time           |  Normal recovery

DR Team and Roles#

DR Command Structure#

text
DR Commander: Overall DR coordination and decisions
├── Technical Recovery Lead: Systems restoration
├── Database Recovery Lead: Data recovery and validation
├── Communications Lead: Stakeholder communication
├── Security Lead: Security validation and compliance
└── Business Continuity Lead: Service prioritization

DR Team Activation#

bash
# Activate DR team via PagerDuty
pagerduty incident create --title="DR Activation - [Scenario]" --service="dr-team"

# Establish DR communication bridge
conference-bridge start dr-emergency

# Notify executive team
alert-executives "DR activation initiated for [scenario]"

DR Role Assignments#

DR Commander#

  • Overall coordination of recovery efforts
  • Resource allocation decisions
  • Recovery priority determination
  • External communication authorization
  • Recovery completion validation

Technical Recovery Lead#

  • Infrastructure restoration procedures
  • Service recovery coordination
  • System validation after recovery
  • Performance testing before full restore

Database Recovery Lead#

  • Data backup validation and restoration
  • Data integrity verification
  • Application data consistency checks
  • Database performance optimization post-recovery

Database Disaster Recovery#

Database Failure Scenarios#

Scenario 1: Primary Database Corruption#

text
Symptoms:
- Database unable to start
- Corruption errors in logs
- Data inconsistency detected
- Backup restore required

Recovery Actions:
1. Isolate corrupted database
2. Validate backup integrity
3. Restore from most recent backup
4. Replay transaction logs if available
5. Validate data consistency
6. Update connection strings to restored DB

Scenario 2: Complete Database Cluster Failure#

text
Symptoms:
- All database nodes unreachable
- Network or storage failure
- Cluster management system down
- No read/write capability

Recovery Actions:
1. Assess infrastructure failure scope
2. Activate backup region database
3. Update DNS/load balancer configuration
4. Validate replication lag and data consistency
5. Redirect application traffic
6. Monitor performance and connection pools

Database Recovery Procedures#

Step 1: Assessment and Isolation#

bash
# Check database cluster status
kubectl get pods -n database -o wide
kubectl describe pods -n database

# Check storage and network
kubectl get pv,pvc -n database
kubectl get services -n database

# Isolate failed database (prevent further damage)
kubectl scale statefulset postgres-primary --replicas=0 -n database

Step 2: Backup Validation#

bash
# List available backups
aws s3 ls s3://lilith-database-backups/ --recursive

# Validate backup integrity
pg_verifybackup --backup-path=/backups/latest

# Check backup metadata
cat /backups/latest/backup_label

Step 3: Backup Restoration#

bash
# Create new database instance
kubectl apply -f database-recovery-pod.yaml

# Restore database from backup
kubectl exec -it database-recovery-pod -- pg_basebackup \
  --host=backup-location --port=5432 \
  --username=postgres --pgdata=/var/lib/postgresql/data

# Apply transaction logs for point-in-time recovery
kubectl exec -it database-recovery-pod -- pg_wal_replay \
  --wal-directory=/backup-wals --target-time="2024-09-15 21:30:00"

Step 4: Data Validation#

bash
# Start database in recovery mode
kubectl exec -it database-recovery-pod -- postgres --config-file=/etc/postgresql/recovery.conf

# Run data consistency checks
kubectl exec -it database-recovery-pod -- psql -c "SELECT count(*) FROM users;"
kubectl exec -it database-recovery-pod -- psql -c "SELECT count(*) FROM conversations;"

# Validate foreign key constraints
kubectl exec -it database-recovery-pod -- psql -c "SELECT * FROM pg_constraint WHERE contype = 'f';"

Step 5: Service Restoration#

bash
# Update service configuration to point to recovered database
kubectl patch configmap database-config --patch='{"data":{"host":"database-recovery-service"}}'

# Restart services to pick up new database connection
kubectl rollout restart deployment bff-deployment -n lilith-bff
kubectl rollout restart deployment auth-deployment -n lilith-auth
kubectl rollout restart deployment conversation-deployment -n lilith-conversation

# Monitor service health
kubectl get pods --all-namespaces | grep -v Running

Service-Level Disaster Recovery#

Multi-Service Failure Recovery#

Scenario: Cascading Service Failures#

text
Symptoms:
- Multiple services failing simultaneously
- Dependency chain failures
- Resource exhaustion across cluster
- Normal scaling/restart not effective

Recovery Strategy:
1. Identify core service dependencies
2. Restore services in dependency order
3. Validate each service before proceeding
4. Monitor for cascading issues

Service Recovery Priority Matrix#

text
Priority 1 (Critical):
- Auth service (required for all other services)
- Database connectivity
- Core networking and DNS

Priority 2 (Core Business):
- BFF (user-facing gateway)
- Conversation service (core functionality)
- Content service (content delivery)

Priority 3 (Supporting):
- Media service
- Notification service
- AI service

Priority 4 (Optional):
- Analytics services
- Monitoring services (if primary failed)
- Development/testing services

Service Recovery Procedures#

Step 1: Infrastructure Assessment#

bash
# Check cluster health
kubectl get nodes -o wide
kubectl get namespaces
kubectl describe nodes | grep -A5 Conditions

# Check resource availability
kubectl top nodes
kubectl describe nodes | grep -A10 "Allocated resources"

# Check critical system pods
kubectl get pods -n kube-system
kubectl get pods -n istio-system

Step 2: Core Service Restoration#

bash
# Restore authentication service first
kubectl apply -f manifests/auth-service.yaml
kubectl wait --for=condition=Ready pod -l app=auth -n lilith-auth --timeout=300s

# Validate auth service
curl -f https://auth.lilith.com/v1/health

# Restore database connectivity
kubectl apply -f manifests/database-proxy.yaml
kubectl wait --for=condition=Ready pod -l app=database-proxy -n database --timeout=300s

Step 3: Business Logic Services#

bash
# Restore BFF (gateway service)
kubectl apply -f manifests/bff-service.yaml
kubectl wait --for=condition=Ready pod -l app=bff -n lilith-bff --timeout=300s

# Restore conversation service
kubectl apply -f manifests/conversation-service.yaml
kubectl wait --for=condition=Ready pod -l app=conversation -n lilith-conversation --timeout=300s

# Validate service connectivity
./scripts/validate-service-mesh.sh

Step 4: Supporting Services#

bash
# Restore remaining services in parallel
kubectl apply -f manifests/media-service.yaml &
kubectl apply -f manifests/notification-service.yaml &
kubectl apply -f manifests/ai-service.yaml &

# Wait for all to be ready
kubectl wait --for=condition=Ready pod -l tier=application --all-namespaces --timeout=600s

Network and Infrastructure Recovery#

Regional Failover Procedures#

Scenario: Complete Regional Failure#

text
Symptoms:
- Primary region completely unreachable
- DNS resolution failures
- Network connectivity lost
- All services in region down

Recovery Actions:
1. Confirm regional failure scope
2. Activate secondary region
3. Update DNS routing
4. Restore data from backups
5. Validate service functionality

Regional Failover Steps#

Step 1: Failure Confirmation#

bash
# Test connectivity from multiple sources
ping primary-region.lilith.com
traceroute primary-region.lilith.com
nslookup primary-region.lilith.com

# Check cloud provider status
aws-status check --region=us-east-1
azure-status check --region=eastus

# Validate scope of failure
kubectl config use-context backup-region
kubectl get nodes

Step 2: DNS and Traffic Routing#

bash
# Update DNS to point to backup region
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://dns-failover.json

# Update load balancer configuration
kubectl patch ingress main-ingress \
  --patch='{"spec":{"rules":[{"host":"lilith.com","http":{"paths":[{"backend":{"service":{"name":"bff-backup","port":{"number":80}}}}]}}]}}'

# Verify DNS propagation
dig lilith.com @8.8.8.8

Step 3: Service Restoration in Backup Region#

bash
# Check backup region cluster status
kubectl config use-context backup-region-cluster
kubectl get nodes
kubectl get namespaces

# Restore services from manifests
kubectl apply -f manifests/backup-region/ --recursive

# Scale services to handle full load
kubectl scale deployment bff-deployment --replicas=10 -n lilith-bff
kubectl scale deployment conversation-deployment --replicas=8 -n lilith-conversation

Step 4: Data Synchronization#

bash
# Restore database from latest backup
kubectl exec -it postgres-backup-0 -n database -- \
  pg_basebackup --host=backup-storage --pgdata=/var/lib/postgresql/data

# Sync media files from backup storage
aws s3 sync s3://lilith-media-backup/ s3://lilith-media-active/ --region=us-west-2

# Validate data consistency
./scripts/validate-data-integrity.sh

Security Incident Recovery#

Security Compromise Recovery#

Scenario: System Compromise Detected#

text
Symptoms:
- Unauthorized access detected
- Malicious code found
- Data exfiltration suspected
- Systems potentially compromised

Recovery Actions:
1. Immediate isolation
2. Evidence preservation
3. Clean system restoration
4. Security validation
5. Monitoring enhancement

Security Recovery Procedures#

Step 1: Immediate Isolation#

bash
# Isolate compromised systems
kubectl patch networkpolicy default-deny --patch='{"spec":{"podSelector":{},"policyTypes":["Ingress","Egress"]}}'

# Block external traffic
kubectl patch ingress main-ingress --patch='{"spec":{"rules":[]}}'

# Preserve evidence
kubectl get pods --all-namespaces -o yaml > incident-pod-state.yaml
kubectl logs --all-containers=true --all-namespaces > incident-logs.txt

Step 2: Clean System Deployment#

bash
# Deploy clean images from secure registry
kubectl set image deployment/bff-deployment bff=lilith-registry/bff:clean-$(date +%Y%m%d)
kubectl set image deployment/auth-deployment auth=lilith-registry/auth:clean-$(date +%Y%m%d)

# Rotate all secrets and certificates
kubectl delete secret --all
kubectl apply -f secrets/clean-secrets.yaml

# Update database passwords
kubectl exec -it postgres-primary-0 -- psql -c "ALTER USER postgres PASSWORD '$(generate-secure-password)';"

Step 3: Security Validation#

bash
# Run security scans on restored systems
trivy image lilith-registry/bff:clean-$(date +%Y%m%d)
./scripts/security-baseline-check.sh

# Validate no backdoors exist
./scripts/malware-scan.sh
./scripts/configuration-audit.sh

DR Validation and Testing#

Recovery Validation Checklist#

System-Level Validation#

  • All critical services responding to health checks
  • Database connectivity and query performance normal
  • Authentication and authorization working
  • Inter-service communication functioning
  • External API integrations operational

User-Level Validation#

  • User login and authentication flows
  • Core application functionality
  • Data persistence and retrieval
  • Real-time features (chat, notifications)
  • File upload and download capabilities

Performance Validation#

  • Response times within acceptable ranges
  • Database query performance normal
  • Resource utilization stable
  • No memory leaks or connection issues
  • Load balancing functioning correctly

DR Testing Schedule#

Monthly Tests#

  • Database backup restoration in test environment
  • Service recovery procedures validation
  • DNS failover testing
  • Communication procedures practice

Quarterly Tests#

  • Full regional failover simulation
  • Multi-service failure recovery
  • Security incident response drill
  • End-to-end DR scenario testing

Annual Tests#

  • Complete disaster simulation with business stakeholders
  • Extended outage scenario (>24 hours)
  • Multiple concurrent failures
  • Vendor dependency failures

DR Communication Procedures#

DR Activation Communication#

Internal Alert#

text
🚨 DISASTER RECOVERY ACTIVATED

Scenario: [Brief description]
Scope: [Affected systems/regions]
DR Commander: @[name]
Estimated Duration: [Initial estimate]

All hands meeting in 15 minutes: [bridge info]
Updates every 30 minutes in #dr-coordination

Do not communicate externally until approved by DR Commander.

Executive Notification#

text
Subject: URGENT - Disaster Recovery Activated - [Scenario]

Executive Summary:
- What: [Business impact description]
- Scope: [Systems and services affected]
- Impact: [Customer and business impact]
- Action: [DR procedures initiated]
- Leadership: [DR Commander and team]

Timeline:
- Incident Start: [Time]
- DR Activation: [Time]
- Estimated Recovery: [Initial estimate]

Business Impact:
- Customer Access: [Status]
- Revenue Impact: [Estimated]
- SLA Impact: [Affected SLAs]

Communication:
- Customer Updates: [Frequency and responsibility]
- Media Response: [Designated spokesperson]
- Regulatory Notifications: [If required]

Next Update: [Time]

Detailed technical updates available via DR Commander.

Customer Communication During DR#

Initial Customer Notification#

text
🔴 SERVICE INTERRUPTION - EMERGENCY MAINTENANCE

We are currently experiencing a significant service interruption affecting multiple systems.

Our team has activated emergency recovery procedures and is working to restore service as quickly as possible.

We understand this is a critical disruption and sincerely apologize for the impact.

Updates will be provided every hour until service is restored.

For urgent inquiries: [emergency contact]
Status updates: [status page]

Next update: [Time]

DR Recovery Progress Update#

text
🟡 RECOVERY IN PROGRESS - Service Restoration Underway

We continue to work on restoring service following [brief description of issue].

Progress:
- [Specific systems restored]
- [Current focus area]
- [Estimated timeline]

Impact:
- [Current service availability]
- [What customers can/cannot access]

We appreciate your continued patience as we work to fully restore service.

Next update: [Time]

Post-DR Review Process#

Immediate Post-DR Actions (Within 24 hours)#

  • Complete system validation and performance testing
  • Document complete timeline of DR activities
  • Preserve all logs and evidence from DR event
  • Schedule post-DR review meeting
  • Update monitoring and alerting based on learnings

DR Review Meeting (Within 1 week)#

  • Timeline Review - Complete chronology of events and decisions
  • Effectiveness Assessment - How well did DR procedures work?
  • Gap Analysis - What procedures were missing or ineffective?
  • Recovery Time Analysis - Where time was lost and could be improved
  • Communication Review - How effective was internal and external communication?

DR Improvement Actions#

  • Procedure Updates - Revise DR playbooks based on learnings
  • Training Updates - Update training materials and conduct refresher training
  • Tool Improvements - Enhance automation and recovery tools
  • Infrastructure Changes - Improve redundancy and recovery capabilities
  • Process Enhancements - Streamline decision-making and communication

Remember: Disaster recovery is about minimizing business impact while ensuring safe, complete restoration. Document everything, communicate regularly, and validate thoroughly before declaring recovery complete.