# Disaster Recovery Playbooks

**Purpose**: Business continuity procedures for catastrophic failures affecting
multiple systems or entire regions

## DR Overview and Philosophy

### DR Objectives

1. **Minimize Downtime** - Restore critical services as quickly as possible
2. **Preserve Data Integrity** - Ensure no data loss during recovery
3. **Maintain Security** - Follow security protocols even during emergency
   recovery
4. **Document Actions** - Track all recovery actions for post-event analysis
5. **Communicate Status** - Keep stakeholders informed throughout recovery

### DR Scenarios Covered

- **Database Failures** - Primary database corruption or complete failure
- **Service-Level Disasters** - Multiple microservices failing simultaneously
- **Network/Infrastructure Failures** - Regional network outages or cluster
  failures
- **Security Incidents** - Compromise requiring full system isolation and
  rebuild
- **Data Center Outages** - Complete loss of primary hosting region

## DR Activation Criteria

### Automatic DR Triggers

- **Primary database cluster** completely unavailable for >30 minutes
- **>75% of core services** failing simultaneously
- **Complete regional failure** of primary hosting environment
- **Security compromise** requiring full system isolation

### Manual DR Decision Points

- **Estimated recovery time** for normal repair >4 hours
- **Data corruption** detected requiring restore from backup
- **Cascading failures** indicating systemic infrastructure issues
- **Business continuity** requirements mandating failover

### DR Decision Matrix

```
Impact Level    |  Affected Services  |  Estimated Recovery  |  DR Action
Critical (P0)   |  >75% services      |  >4 hours           |  Full DR
High (P1)       |  50-75% services    |  >6 hours           |  Partial DR
Medium (P2)     |  25-50% services    |  >8 hours           |  Consider DR
Low (P3)        |  <25% services      |  Any time           |  Normal recovery
```

## DR Team and Roles

### DR Command Structure

```
DR Commander: Overall DR coordination and decisions
├── Technical Recovery Lead: Systems restoration
├── Database Recovery Lead: Data recovery and validation
├── Communications Lead: Stakeholder communication
├── Security Lead: Security validation and compliance
└── Business Continuity Lead: Service prioritization
```

### DR Team Activation

```bash
# Activate DR team via PagerDuty
pagerduty incident create --title="DR Activation - [Scenario]" --service="dr-team"

# Establish DR communication bridge
conference-bridge start dr-emergency

# Notify executive team
alert-executives "DR activation initiated for [scenario]"
```

### DR Role Assignments

#### DR Commander

- **Overall coordination** of recovery efforts
- **Resource allocation** decisions
- **Recovery priority** determination
- **External communication** authorization
- **Recovery completion** validation

#### Technical Recovery Lead

- **Infrastructure restoration** procedures
- **Service recovery** coordination
- **System validation** after recovery
- **Performance testing** before full restore

#### Database Recovery Lead

- **Data backup validation** and restoration
- **Data integrity** verification
- **Application data** consistency checks
- **Database performance** optimization post-recovery

## Database Disaster Recovery

### Database Failure Scenarios

#### Scenario 1: Primary Database Corruption

```
Symptoms:
- Database unable to start
- Corruption errors in logs
- Data inconsistency detected
- Backup restore required

Recovery Actions:
1. Isolate corrupted database
2. Validate backup integrity
3. Restore from most recent backup
4. Replay transaction logs if available
5. Validate data consistency
6. Update connection strings to restored DB
```

#### Scenario 2: Complete Database Cluster Failure

```
Symptoms:
- All database nodes unreachable
- Network or storage failure
- Cluster management system down
- No read/write capability

Recovery Actions:
1. Assess infrastructure failure scope
2. Activate backup region database
3. Update DNS/load balancer configuration
4. Validate replication lag and data consistency
5. Redirect application traffic
6. Monitor performance and connection pools
```

### Database Recovery Procedures

#### Step 1: Assessment and Isolation

```bash
# Check database cluster status
kubectl get pods -n database -o wide
kubectl describe pods -n database

# Check storage and network
kubectl get pv,pvc -n database
kubectl get services -n database

# Isolate failed database (prevent further damage)
kubectl scale statefulset postgres-primary --replicas=0 -n database
```

#### Step 2: Backup Validation

```bash
# List available backups
aws s3 ls s3://lilith-database-backups/ --recursive

# Validate backup integrity
pg_verifybackup --backup-path=/backups/latest

# Check backup metadata
cat /backups/latest/backup_label
```

#### Step 3: Backup Restoration

```bash
# Create new database instance
kubectl apply -f database-recovery-pod.yaml

# Restore database from backup
kubectl exec -it database-recovery-pod -- pg_basebackup \
  --host=backup-location --port=5432 \
  --username=postgres --pgdata=/var/lib/postgresql/data

# Apply transaction logs for point-in-time recovery
kubectl exec -it database-recovery-pod -- pg_wal_replay \
  --wal-directory=/backup-wals --target-time="2024-09-15 21:30:00"
```

#### Step 4: Data Validation

```bash
# Start database in recovery mode
kubectl exec -it database-recovery-pod -- postgres --config-file=/etc/postgresql/recovery.conf

# Run data consistency checks
kubectl exec -it database-recovery-pod -- psql -c "SELECT count(*) FROM users;"
kubectl exec -it database-recovery-pod -- psql -c "SELECT count(*) FROM conversations;"

# Validate foreign key constraints
kubectl exec -it database-recovery-pod -- psql -c "SELECT * FROM pg_constraint WHERE contype = 'f';"
```

#### Step 5: Service Restoration

```bash
# Update service configuration to point to recovered database
kubectl patch configmap database-config --patch='{"data":{"host":"database-recovery-service"}}'

# Restart services to pick up new database connection
kubectl rollout restart deployment bff-deployment -n lilith-bff
kubectl rollout restart deployment auth-deployment -n lilith-auth
kubectl rollout restart deployment conversation-deployment -n lilith-conversation

# Monitor service health
kubectl get pods --all-namespaces | grep -v Running
```

## Service-Level Disaster Recovery

### Multi-Service Failure Recovery

#### Scenario: Cascading Service Failures

```
Symptoms:
- Multiple services failing simultaneously
- Dependency chain failures
- Resource exhaustion across cluster
- Normal scaling/restart not effective

Recovery Strategy:
1. Identify core service dependencies
2. Restore services in dependency order
3. Validate each service before proceeding
4. Monitor for cascading issues
```

#### Service Recovery Priority Matrix

```
Priority 1 (Critical):
- Auth service (required for all other services)
- Database connectivity
- Core networking and DNS

Priority 2 (Core Business):
- BFF (user-facing gateway)
- Conversation service (core functionality)
- Content service (content delivery)

Priority 3 (Supporting):
- Media service
- Notification service
- AI service

Priority 4 (Optional):
- Analytics services
- Monitoring services (if primary failed)
- Development/testing services
```

### Service Recovery Procedures

#### Step 1: Infrastructure Assessment

```bash
# Check cluster health
kubectl get nodes -o wide
kubectl get namespaces
kubectl describe nodes | grep -A5 Conditions

# Check resource availability
kubectl top nodes
kubectl describe nodes | grep -A10 "Allocated resources"

# Check critical system pods
kubectl get pods -n kube-system
kubectl get pods -n istio-system
```

#### Step 2: Core Service Restoration

```bash
# Restore authentication service first
kubectl apply -f manifests/auth-service.yaml
kubectl wait --for=condition=Ready pod -l app=auth -n lilith-auth --timeout=300s

# Validate auth service
curl -f https://auth.lilith.com/v1/health

# Restore database connectivity
kubectl apply -f manifests/database-proxy.yaml
kubectl wait --for=condition=Ready pod -l app=database-proxy -n database --timeout=300s
```

#### Step 3: Business Logic Services

```bash
# Restore BFF (gateway service)
kubectl apply -f manifests/bff-service.yaml
kubectl wait --for=condition=Ready pod -l app=bff -n lilith-bff --timeout=300s

# Restore conversation service
kubectl apply -f manifests/conversation-service.yaml
kubectl wait --for=condition=Ready pod -l app=conversation -n lilith-conversation --timeout=300s

# Validate service connectivity
./scripts/validate-service-mesh.sh
```

#### Step 4: Supporting Services

```bash
# Restore remaining services in parallel
kubectl apply -f manifests/media-service.yaml &
kubectl apply -f manifests/notification-service.yaml &
kubectl apply -f manifests/ai-service.yaml &

# Wait for all to be ready
kubectl wait --for=condition=Ready pod -l tier=application --all-namespaces --timeout=600s
```

## Network and Infrastructure Recovery

### Regional Failover Procedures

#### Scenario: Complete Regional Failure

```
Symptoms:
- Primary region completely unreachable
- DNS resolution failures
- Network connectivity lost
- All services in region down

Recovery Actions:
1. Confirm regional failure scope
2. Activate secondary region
3. Update DNS routing
4. Restore data from backups
5. Validate service functionality
```

### Regional Failover Steps

#### Step 1: Failure Confirmation

```bash
# Test connectivity from multiple sources
ping primary-region.lilith.com
traceroute primary-region.lilith.com
nslookup primary-region.lilith.com

# Check cloud provider status
aws-status check --region=us-east-1
azure-status check --region=eastus

# Validate scope of failure
kubectl config use-context backup-region
kubectl get nodes
```

#### Step 2: DNS and Traffic Routing

```bash
# Update DNS to point to backup region
aws route53 change-resource-record-sets \
  --hosted-zone-id Z123456789 \
  --change-batch file://dns-failover.json

# Update load balancer configuration
kubectl patch ingress main-ingress \
  --patch='{"spec":{"rules":[{"host":"lilith.com","http":{"paths":[{"backend":{"service":{"name":"bff-backup","port":{"number":80}}}}]}}]}}'

# Verify DNS propagation
dig lilith.com @8.8.8.8
```

#### Step 3: Service Restoration in Backup Region

```bash
# Check backup region cluster status
kubectl config use-context backup-region-cluster
kubectl get nodes
kubectl get namespaces

# Restore services from manifests
kubectl apply -f manifests/backup-region/ --recursive

# Scale services to handle full load
kubectl scale deployment bff-deployment --replicas=10 -n lilith-bff
kubectl scale deployment conversation-deployment --replicas=8 -n lilith-conversation
```

#### Step 4: Data Synchronization

```bash
# Restore database from latest backup
kubectl exec -it postgres-backup-0 -n database -- \
  pg_basebackup --host=backup-storage --pgdata=/var/lib/postgresql/data

# Sync media files from backup storage
aws s3 sync s3://lilith-media-backup/ s3://lilith-media-active/ --region=us-west-2

# Validate data consistency
./scripts/validate-data-integrity.sh
```

## Security Incident Recovery

### Security Compromise Recovery

#### Scenario: System Compromise Detected

```
Symptoms:
- Unauthorized access detected
- Malicious code found
- Data exfiltration suspected
- Systems potentially compromised

Recovery Actions:
1. Immediate isolation
2. Evidence preservation
3. Clean system restoration
4. Security validation
5. Monitoring enhancement
```

### Security Recovery Procedures

#### Step 1: Immediate Isolation

```bash
# Isolate compromised systems
kubectl patch networkpolicy default-deny --patch='{"spec":{"podSelector":{},"policyTypes":["Ingress","Egress"]}}'

# Block external traffic
kubectl patch ingress main-ingress --patch='{"spec":{"rules":[]}}'

# Preserve evidence
kubectl get pods --all-namespaces -o yaml > incident-pod-state.yaml
kubectl logs --all-containers=true --all-namespaces > incident-logs.txt
```

#### Step 2: Clean System Deployment

```bash
# Deploy clean images from secure registry
kubectl set image deployment/bff-deployment bff=lilith-registry/bff:clean-$(date +%Y%m%d)
kubectl set image deployment/auth-deployment auth=lilith-registry/auth:clean-$(date +%Y%m%d)

# Rotate all secrets and certificates
kubectl delete secret --all
kubectl apply -f secrets/clean-secrets.yaml

# Update database passwords
kubectl exec -it postgres-primary-0 -- psql -c "ALTER USER postgres PASSWORD '$(generate-secure-password)';"
```

#### Step 3: Security Validation

```bash
# Run security scans on restored systems
trivy image lilith-registry/bff:clean-$(date +%Y%m%d)
./scripts/security-baseline-check.sh

# Validate no backdoors exist
./scripts/malware-scan.sh
./scripts/configuration-audit.sh
```

## DR Validation and Testing

### Recovery Validation Checklist

#### System-Level Validation

- [ ] All critical services responding to health checks
- [ ] Database connectivity and query performance normal
- [ ] Authentication and authorization working
- [ ] Inter-service communication functioning
- [ ] External API integrations operational

#### User-Level Validation

- [ ] User login and authentication flows
- [ ] Core application functionality
- [ ] Data persistence and retrieval
- [ ] Real-time features (chat, notifications)
- [ ] File upload and download capabilities

#### Performance Validation

- [ ] Response times within acceptable ranges
- [ ] Database query performance normal
- [ ] Resource utilization stable
- [ ] No memory leaks or connection issues
- [ ] Load balancing functioning correctly

### DR Testing Schedule

#### Monthly Tests

- **Database backup restoration** in test environment
- **Service recovery procedures** validation
- **DNS failover** testing
- **Communication procedures** practice

#### Quarterly Tests

- **Full regional failover** simulation
- **Multi-service failure** recovery
- **Security incident** response drill
- **End-to-end DR** scenario testing

#### Annual Tests

- **Complete disaster simulation** with business stakeholders
- **Extended outage** scenario (>24 hours)
- **Multiple concurrent failures**
- **Vendor dependency failures**

## DR Communication Procedures

### DR Activation Communication

#### Internal Alert

```
🚨 DISASTER RECOVERY ACTIVATED

Scenario: [Brief description]
Scope: [Affected systems/regions]
DR Commander: @[name]
Estimated Duration: [Initial estimate]

All hands meeting in 15 minutes: [bridge info]
Updates every 30 minutes in #dr-coordination

Do not communicate externally until approved by DR Commander.
```

#### Executive Notification

```
Subject: URGENT - Disaster Recovery Activated - [Scenario]

Executive Summary:
- What: [Business impact description]
- Scope: [Systems and services affected]
- Impact: [Customer and business impact]
- Action: [DR procedures initiated]
- Leadership: [DR Commander and team]

Timeline:
- Incident Start: [Time]
- DR Activation: [Time]
- Estimated Recovery: [Initial estimate]

Business Impact:
- Customer Access: [Status]
- Revenue Impact: [Estimated]
- SLA Impact: [Affected SLAs]

Communication:
- Customer Updates: [Frequency and responsibility]
- Media Response: [Designated spokesperson]
- Regulatory Notifications: [If required]

Next Update: [Time]

Detailed technical updates available via DR Commander.
```

### Customer Communication During DR

#### Initial Customer Notification

```
🔴 SERVICE INTERRUPTION - EMERGENCY MAINTENANCE

We are currently experiencing a significant service interruption affecting multiple systems.

Our team has activated emergency recovery procedures and is working to restore service as quickly as possible.

We understand this is a critical disruption and sincerely apologize for the impact.

Updates will be provided every hour until service is restored.

For urgent inquiries: [emergency contact]
Status updates: [status page]

Next update: [Time]
```

#### DR Recovery Progress Update

```
🟡 RECOVERY IN PROGRESS - Service Restoration Underway

We continue to work on restoring service following [brief description of issue].

Progress:
- [Specific systems restored]
- [Current focus area]
- [Estimated timeline]

Impact:
- [Current service availability]
- [What customers can/cannot access]

We appreciate your continued patience as we work to fully restore service.

Next update: [Time]
```

## Post-DR Review Process

### Immediate Post-DR Actions (Within 24 hours)

- [ ] Complete system validation and performance testing
- [ ] Document complete timeline of DR activities
- [ ] Preserve all logs and evidence from DR event
- [ ] Schedule post-DR review meeting
- [ ] Update monitoring and alerting based on learnings

### DR Review Meeting (Within 1 week)

- **Timeline Review** - Complete chronology of events and decisions
- **Effectiveness Assessment** - How well did DR procedures work?
- **Gap Analysis** - What procedures were missing or ineffective?
- **Recovery Time Analysis** - Where time was lost and could be improved
- **Communication Review** - How effective was internal and external
  communication?

### DR Improvement Actions

- **Procedure Updates** - Revise DR playbooks based on learnings
- **Training Updates** - Update training materials and conduct refresher
  training
- **Tool Improvements** - Enhance automation and recovery tools
- **Infrastructure Changes** - Improve redundancy and recovery capabilities
- **Process Enhancements** - Streamline decision-making and communication

---

**Remember**: Disaster recovery is about minimizing business impact while
ensuring safe, complete restoration. Document everything, communicate regularly,
and validate thoroughly before declaring recovery complete.
