# Psyche Domain: Operations Runbooks

This document provides operational runbooks for the Psyche (Serwaa) AI Virtual
Assistant Platform, covering deployment procedures, GPU troubleshooting,
performance tuning, and incident response.

## Table of Contents

1. [Deployment Procedures](#deployment-procedures)
2. [GPU Troubleshooting](#gpu-troubleshooting)
3. [Performance Tuning](#performance-tuning)
4. [Incident Response](#incident-response)
5. [Common Operations](#common-operations)
6. [Monitoring & Alerting](#monitoring--alerting)

---

## Deployment Procedures

### 1. Pre-Deployment Checklist

Before deploying any Psyche service:

```bash
# 1. Verify cluster health
kubectl get nodes -o wide
kubectl get pods -n serwaa --field-selector=status.phase!=Running

# 2. Check GPU availability
kubectl describe nodes | grep -A5 "nvidia.com/gpu"

# 3. Verify secrets exist
kubectl get secrets -n serwaa | grep -E "(db-credentials|api-keys|tls)"

# 4. Check persistent volumes
kubectl get pv,pvc -n serwaa

# 5. Verify model registry access
aws s3 ls s3://serwaa-models/registry/index.json
```

### 2. Deploying a New Service Version

#### 2.1 Rolling Deployment (Default)

```bash
# Update image tag in values.yaml or use --set
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
  --namespace serwaa \
  --set image.tag=v1.2.3 \
  --set image.pullPolicy=Always \
  --wait \
  --timeout 10m

# Monitor rollout
kubectl rollout status deployment/serwaa-avatar-engine -n serwaa

# Verify new pods are running
kubectl get pods -n serwaa -l app=avatar-engine -o wide
```

#### 2.2 Blue-Green Deployment

```bash
# Deploy new version as "green" alongside existing "blue"
helm install serwaa-avatar-engine-green ./charts/avatar-engine \
  --namespace serwaa \
  --set image.tag=v1.2.3 \
  --set service.name=avatar-engine-green \
  --set replicaCount=3

# Run smoke tests against green deployment
kubectl run smoke-test --rm -it --image=curlimages/curl -- \
  curl -s http://avatar-engine-green:8001/health

# Switch traffic (update ingress or service selector)
kubectl patch service avatar-engine -n serwaa \
  -p '{"spec":{"selector":{"version":"green"}}}'

# Verify traffic is flowing to green
kubectl logs -f -l app=avatar-engine,version=green -n serwaa | head -50

# Cleanup blue deployment after verification
helm uninstall serwaa-avatar-engine-blue -n serwaa
```

#### 2.3 Canary Deployment

```bash
# Deploy canary with 10% traffic
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
  --namespace serwaa \
  --set canary.enabled=true \
  --set canary.weight=10 \
  --set canary.image.tag=v1.2.3

# Monitor canary metrics
kubectl get virtualservice avatar-engine -n serwaa -o yaml

# Gradually increase canary weight
for weight in 25 50 75 100; do
  helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
    --set canary.weight=$weight
  echo "Canary weight: $weight% - waiting 5 minutes..."
  sleep 300
done
```

### 3. Rolling Back a Deployment

```bash
# Check rollout history
kubectl rollout history deployment/serwaa-avatar-engine -n serwaa

# Rollback to previous version
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa

# Rollback to specific revision
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa --to-revision=3

# If using Helm
helm rollback serwaa-avatar-engine 1 -n serwaa
```

### 4. Model Deployment

#### 4.1 Promoting a Model to Production

```bash
# Validate model in staging
curl -X POST https://staging.api.serwaa.ai/api/v1/models/validate \
  -H "Authorization: Bearer $API_TOKEN" \
  -d '{"model_name": "3dgs-avatar", "version": "1.3.0"}'

# Promote to production
aws s3 cp s3://serwaa-models-staging/avatar/3dgs/v1.3.0/ \
          s3://serwaa-models/avatar/3dgs/v1.3.0/ --recursive

# Update model registry index
python scripts/update-model-registry.py \
  --model 3dgs-avatar \
  --version 1.3.0 \
  --status production

# Trigger hot-reload in services
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa
```

#### 4.2 Rolling Back a Model

```bash
# Identify previous version
aws s3 ls s3://serwaa-models/avatar/3dgs/

# Update current version pointer
aws s3 cp s3://serwaa-models/avatar/3dgs/v1.2.0/metadata.json \
          s3://serwaa-models/avatar/3dgs/current/metadata.json

# Restart services to pick up change
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa

# Verify model version in service
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  curl localhost:8001/models/active
```

---

## GPU Troubleshooting

### 1. GPU Not Detected

**Symptoms:**

- Pod stuck in Pending state
- CUDA initialization errors in logs

**Diagnosis:**

```bash
# Check if GPU nodes are available
kubectl get nodes -l nvidia.com/gpu.present=true

# Check GPU allocatable resources
kubectl describe node <gpu-node> | grep -A10 "Allocatable:"

# Check NVIDIA device plugin status
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds

# Check device plugin logs
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds
```

**Resolution:**

```bash
# If device plugin is not running
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml

# If node needs driver update
ssh admin@<gpu-node>
sudo apt update && sudo apt install nvidia-driver-535

# Restart kubelet after driver update
sudo systemctl restart kubelet
```

### 2. CUDA Out of Memory (OOM)

**Symptoms:**

- `CUDA out of memory` errors
- Pod killed with OOMKilled status
- Service crashes during inference

**Diagnosis:**

```bash
# Check GPU memory usage
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi

# Check pod resource limits
kubectl describe pod <pod-name> -n serwaa | grep -A5 "Limits:"

# Monitor GPU memory over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  watch -n 1 'nvidia-smi --query-gpu=memory.used,memory.total --format=csv'
```

**Resolution:**

```bash
# 1. Increase GPU memory limits (if available)
kubectl patch deployment serwaa-avatar-engine -n serwaa \
  --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/nvidia.com~1gpu", "value": "2"}]'

# 2. Enable memory optimization in service config
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:512" \
  TORCH_CUDNN_V8_API_ENABLED=1

# 3. Reduce batch size or model precision
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  MODEL_PRECISION=fp16 \
  MAX_BATCH_SIZE=1

# 4. Clear GPU cache manually (within pod)
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; torch.cuda.empty_cache()"
```

### 3. GPU Thermal Throttling

**Symptoms:**

- Inference latency gradually increasing
- GPU clock speed reduced
- High GPU temperature warnings

**Diagnosis:**

```bash
# Check GPU temperature
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks.current.graphics --format=csv

# Monitor over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  nvidia-smi dmon -s puc -d 5
```

**Resolution:**

```bash
# 1. Check cooling system on physical nodes
ssh admin@<gpu-node>
sudo ipmitool sensor list | grep -i fan
sudo ipmitool chassis status

# 2. Reduce power limit temporarily
ssh admin@<gpu-node>
sudo nvidia-smi -pl 200  # Reduce from default 350W

# 3. Spread workload across multiple GPUs
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4
```

### 4. CUDA Driver Mismatch

**Symptoms:**

- `CUDA driver version is insufficient for CUDA runtime version`
- Services fail to start

**Diagnosis:**

```bash
# Check driver version on node
ssh admin@<gpu-node>
nvidia-smi --query-gpu=driver_version --format=csv

# Check CUDA version required by container
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; print(torch.version.cuda)"
```

**Resolution:**

```bash
# Update NVIDIA driver on node
ssh admin@<gpu-node>
sudo apt update
sudo apt install nvidia-driver-535  # Match container requirements
sudo reboot

# Or use older container image with matching CUDA
kubectl set image deployment/serwaa-avatar-engine -n serwaa \
  avatar-engine=ghcr.io/iron-forge/serwaa/avatar-engine:v1.2.0-cuda11.8
```

---

## Performance Tuning

### 1. Avatar Rendering Optimization

#### 1.1 Latency Reduction

```bash
# Enable TensorRT for faster inference
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  USE_TENSORRT=true \
  TENSORRT_PRECISION=fp16

# Reduce Gaussian count for faster rendering
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  MAX_GAUSSIANS=75000 \
  SH_DEGREE=2

# Enable CUDA graph optimization
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  ENABLE_CUDA_GRAPHS=true \
  CUDA_GRAPH_WARMUP_ITERATIONS=100
```

#### 1.2 Throughput Improvement

```bash
# Enable batching
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  ENABLE_BATCHING=true \
  MAX_BATCH_SIZE=4 \
  BATCH_TIMEOUT_MS=5

# Scale replicas based on GPU availability
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=3

# Configure HPA for auto-scaling
kubectl apply -f - <<EOF
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: avatar-engine-hpa
  namespace: serwaa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: serwaa-avatar-engine
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: nvidia.com/gpu
      target:
        type: Utilization
        averageUtilization: 70
EOF
```

### 2. Voice Engine Optimization

```bash
# Enable streaming mode for lower latency
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  TTS_STREAMING_ENABLED=true \
  TTS_CHUNK_SIZE_MS=100

# Use INT8 quantization for Whisper
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  WHISPER_QUANTIZATION=int8 \
  WHISPER_BATCH_SIZE=1

# Enable voice caching
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  VOICE_CACHE_ENABLED=true \
  VOICE_CACHE_MAX_SIZE_MB=1024
```

### 3. Database Optimization

```bash
# PostgreSQL performance tuning
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  ALTER SYSTEM SET shared_buffers = '4GB';
  ALTER SYSTEM SET effective_cache_size = '12GB';
  ALTER SYSTEM SET work_mem = '256MB';
  ALTER SYSTEM SET maintenance_work_mem = '1GB';
  ALTER SYSTEM SET random_page_cost = 1.1;
  ALTER SYSTEM SET effective_io_concurrency = 200;
  ALTER SYSTEM SET max_worker_processes = 8;
  ALTER SYSTEM SET max_parallel_workers_per_gather = 4;
  SELECT pg_reload_conf();
"

# Analyze and vacuum tables
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  ANALYZE;
  VACUUM ANALYZE sessions;
  VACUUM ANALYZE personas;
  VACUUM ANALYZE knowledge_documents;
"

# Create missing indexes
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_status
    ON sessions(status) WHERE status IN ('active', 'pending');
  CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_persona
    ON sessions(persona_id, created_at DESC);
"
```

### 4. Redis Optimization

```bash
# Check memory usage
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli INFO memory

# Adjust memory policy
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli CONFIG SET \
  maxmemory 2gb \
  maxmemory-policy allkeys-lru

# Enable compression for large values
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  REDIS_COMPRESSION=lz4 \
  REDIS_COMPRESSION_THRESHOLD_BYTES=1024
```

### 5. Network Optimization

```bash
# Enable HTTP/2 for gRPC
kubectl annotate ingress serwaa-ingress -n serwaa \
  nginx.ingress.kubernetes.io/backend-protocol="GRPC"

# Configure connection pooling
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  HTTP_POOL_SIZE=100 \
  HTTP_POOL_TIMEOUT_MS=30000 \
  GRPC_POOL_SIZE=50
```

---

## Incident Response

### 1. Incident Severity Levels

| Level   | Impact                  | Response Time | Examples                          |
| ------- | ----------------------- | ------------- | --------------------------------- |
| P1/SEV1 | Complete outage         | 15 minutes    | All sessions failing, data loss   |
| P2/SEV2 | Significant degradation | 1 hour        | 50%+ latency increase, 10% errors |
| P3/SEV3 | Minor issues            | 4 hours       | Single service degraded           |
| P4/SEV4 | Minimal impact          | 24 hours      | Non-critical bug, cosmetic        |

### 2. Incident Response Playbook

#### 2.1 Initial Assessment (First 5 minutes)

```bash
# 1. Check overall system health
kubectl get pods -n serwaa --field-selector=status.phase!=Running

# 2. Check recent events
kubectl get events -n serwaa --sort-by='.lastTimestamp' | tail -20

# 3. Check service health endpoints
for svc in api-gateway avatar-engine voice-engine; do
  echo "=== $svc ==="
  kubectl exec -it deploy/serwaa-$svc -n serwaa -- curl -s localhost:8000/health
done

# 4. Check error rates in metrics
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..'}[5m])"

# 5. Check recent logs for errors
kubectl logs -l app.kubernetes.io/instance=serwaa -n serwaa --since=5m | grep -i error | tail -50
```

#### 2.2 Communication Template

```
INCIDENT DECLARED: [P1/P2/P3/P4]

Time: [UTC timestamp]
Impact: [Brief description of user impact]
Affected Services: [List of services]
Status: [Investigating/Identified/Mitigated/Resolved]

Current Actions:
- [Action 1]
- [Action 2]

Next Update: [Time of next update]

Incident Commander: [Name]
Communications: [#serwaa-incidents Slack channel]
```

#### 2.3 P1 Response: Complete Outage

```bash
# Step 1: Identify failing components
kubectl get pods -n serwaa -o wide
kubectl get svc -n serwaa

# Step 2: Check for resource exhaustion
kubectl top pods -n serwaa
kubectl top nodes

# Step 3: Check external dependencies
# Database
kubectl exec -it serwaa-postgresql-0 -n serwaa -- pg_isready
# Redis
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli ping
# S3
aws s3 ls s3://serwaa-models/ --max-items 1

# Step 4: If database is down
kubectl rollout restart statefulset/serwaa-postgresql -n serwaa

# Step 5: If application pods are crashing
kubectl describe pod <crashing-pod> -n serwaa
kubectl logs <crashing-pod> -n serwaa --previous

# Step 6: Emergency rollback if recent deployment
helm rollback serwaa-api-gateway -n serwaa
helm rollback serwaa-avatar-engine -n serwaa

# Step 7: Scale up for capacity issues
kubectl scale deployment/serwaa-api-gateway -n serwaa --replicas=5
```

#### 2.4 P2 Response: Performance Degradation

```bash
# Step 1: Identify slow services
kubectl top pods -n serwaa --containers | sort -k4 -rn | head -10

# Step 2: Check latency metrics
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(http_request_duration_seconds_bucket[5m]))"

# Step 3: Check for hotspots
kubectl exec -it deploy/serwaa-api-gateway -n serwaa -- \
  cat /tmp/profiling/hotspots.log

# Step 4: Check GPU utilization
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi

# Step 5: Increase capacity
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4

# Step 6: Enable rate limiting to protect system
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  RATE_LIMIT_ENABLED=true \
  RATE_LIMIT_REQUESTS_PER_MINUTE=50
```

### 3. Post-Incident Actions

```bash
# 1. Collect diagnostic data
kubectl cluster-info dump --output-directory=/tmp/incident-$(date +%Y%m%d-%H%M%S)

# 2. Export relevant logs
kubectl logs -l app=serwaa -n serwaa --since=2h > incident-logs.txt

# 3. Export metrics
curl -s "http://prometheus:9090/api/v1/query_range?query=up&start=$(date -d '2 hours ago' +%s)&end=$(date +%s)&step=60" > metrics-export.json

# 4. Generate incident report
cat > incident-report.md <<EOF
# Incident Report: [INCIDENT-YYYY-MM-DD-XXX]

## Summary
[Brief description]

## Timeline
- [HH:MM UTC] - Incident detected
- [HH:MM UTC] - Incident declared
- [HH:MM UTC] - Root cause identified
- [HH:MM UTC] - Mitigation applied
- [HH:MM UTC] - Incident resolved

## Root Cause
[Detailed explanation]

## Impact
- Duration: X hours Y minutes
- Affected users: N
- Failed requests: N

## Action Items
- [ ] [Action 1]
- [ ] [Action 2]

## Lessons Learned
[What we learned]
EOF
```

---

## Common Operations

### 1. Scaling Services

```bash
# Manual scaling
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=5

# Check current replicas
kubectl get deployment -n serwaa -o wide

# Configure HPA
kubectl autoscale deployment/serwaa-avatar-engine -n serwaa \
  --min=2 --max=10 --cpu-percent=70
```

### 2. Viewing Logs

```bash
# Stream logs from all pods of a service
kubectl logs -f -l app=avatar-engine -n serwaa --all-containers

# Search for specific patterns
kubectl logs -l app=api-gateway -n serwaa --since=1h | grep -i "error\|exception"

# Export logs to file
kubectl logs deploy/serwaa-api-gateway -n serwaa --since=24h > api-gateway-logs.txt

# Tail last N lines
kubectl logs deploy/serwaa-voice-engine -n serwaa --tail=100
```

### 3. Debugging Pods

```bash
# Get shell access
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- /bin/bash

# Run diagnostics inside pod
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0)}')"

# Copy files from pod
kubectl cp serwaa/serwaa-avatar-engine-xxx:/tmp/debug.log ./debug.log

# Port-forward for local access
kubectl port-forward deploy/serwaa-api-gateway -n serwaa 8000:8000
```

### 4. Certificate Management

```bash
# Check certificate expiry
kubectl get certificate -n serwaa
kubectl describe certificate serwaa-tls -n serwaa

# Manually trigger renewal
kubectl delete certificate serwaa-tls -n serwaa
kubectl apply -f certificates/serwaa-tls.yaml

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager
```

### 5. Secret Rotation

```bash
# Rotate database password
NEW_PASSWORD=$(openssl rand -base64 32)
kubectl create secret generic db-credentials \
  --from-literal=password=$NEW_PASSWORD \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart services to pick up new secrets
kubectl rollout restart deployment -n serwaa

# Rotate API keys
kubectl patch secret api-keys -n serwaa \
  --patch="{\"stringData\":{\"anthropic-api-key\":\"$NEW_API_KEY\"}}"
```

---

## Monitoring & Alerting

### 1. Key Metrics to Monitor

| Metric                        | Warning Threshold | Critical Threshold |
| ----------------------------- | ----------------- | ------------------ |
| API Gateway Error Rate        | > 1%              | > 5%               |
| Avatar Render Latency (p99)   | > 50ms            | > 100ms            |
| Voice Synthesis Latency (p99) | > 150ms           | > 300ms            |
| GPU Memory Utilization        | > 80%             | > 95%              |
| GPU Temperature               | > 75°C            | > 85°C             |
| Active Sessions               | > 80% capacity    | > 95% capacity     |
| Database Connections          | > 80% max         | > 95% max          |
| Redis Memory                  | > 80% max         | > 95% max          |

### 2. Alert Configuration

```yaml
# prometheus-alerts.yaml
groups:
  - name: serwaa-alerts
    rules:
      - alert: HighErrorRate
        expr:
          rate(http_requests_total{status=~"5.."}[5m]) /
          rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'High error rate in {{ $labels.service }}'

      - alert: HighLatency
        expr:
          histogram_quantile(0.99,
          rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High latency in {{ $labels.service }}'

      - alert: GPUMemoryHigh
        expr:
          nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'GPU memory exhausted on {{ $labels.instance }}'

      - alert: GPUTemperatureHigh
        expr: nvidia_gpu_temperature_celsius > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'GPU overheating on {{ $labels.instance }}'
```

### 3. Grafana Dashboards

Access Grafana at `https://grafana.serwaa.ai` (or port-forward locally):

```bash
kubectl port-forward svc/grafana -n monitoring 3000:80
```

**Key Dashboards:**

- **Serwaa Overview**: High-level system health
- **Avatar Engine**: GPU metrics, render latency, FPS
- **Voice Engine**: TTS/STT latency, success rates
- **API Gateway**: Request rates, error rates, latencies
- **Database**: Connection pools, query performance
- **Redis**: Memory usage, hit rates, latency

### 4. Log Aggregation

```bash
# Access Kibana
kubectl port-forward svc/kibana -n logging 5601:5601

# Query logs via CLI
kubectl exec -it deploy/elasticsearch -n logging -- \
  curl -s "localhost:9200/serwaa-*/_search?q=level:error&size=10" | jq

# Create log-based alerts
curl -X PUT "http://elasticsearch:9200/_watcher/watch/high-error-rate" \
  -H "Content-Type: application/json" \
  -d @error-rate-watch.json
```

---

## Emergency Contacts

| Role                | Contact               | Escalation Time  |
| ------------------- | --------------------- | ---------------- |
| On-Call Engineer    | PagerDuty: serwaa-eng | Immediate        |
| Platform Lead       | [Email/Phone]         | 15 minutes       |
| Engineering Manager | [Email/Phone]         | 30 minutes       |
| VP Engineering      | [Email/Phone]         | 1 hour (P1 only) |

---

## Related Documentation

- [Architecture Overview](./architecture.md)
- [API Documentation](./api-documentation.md)
- [ML Models](./ml-models.md)
- [Performance Baseline](./performance-baseline.md)

---

## Revision History

| Version | Date       | Author      | Changes          |
| ------- | ---------- | ----------- | ---------------- |
| 1.0.0   | 2026-01-31 | Claude Opus | Initial runbooks |
