Eve library

Psyche Domain: Operations Runbooks

Guide

Revision 1

On this page

This document provides operational runbooks for the Psyche (Serwaa) AI Virtual Assistant Platform, covering deployment procedures, GPU troubleshooting, performance tuning, and incident response.

Table of Contents#

  1. Deployment Procedures
  2. GPU Troubleshooting
  3. Performance Tuning
  4. Incident Response
  5. Common Operations
  6. Monitoring & Alerting

Deployment Procedures#

1. Pre-Deployment Checklist#

Before deploying any Psyche service:

bash
# 1. Verify cluster health
kubectl get nodes -o wide
kubectl get pods -n serwaa --field-selector=status.phase!=Running

# 2. Check GPU availability
kubectl describe nodes | grep -A5 "nvidia.com/gpu"

# 3. Verify secrets exist
kubectl get secrets -n serwaa | grep -E "(db-credentials|api-keys|tls)"

# 4. Check persistent volumes
kubectl get pv,pvc -n serwaa

# 5. Verify model registry access
aws s3 ls s3://serwaa-models/registry/index.json

2. Deploying a New Service Version#

2.1 Rolling Deployment (Default)#

bash
# Update image tag in values.yaml or use --set
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
  --namespace serwaa \
  --set image.tag=v1.2.3 \
  --set image.pullPolicy=Always \
  --wait \
  --timeout 10m

# Monitor rollout
kubectl rollout status deployment/serwaa-avatar-engine -n serwaa

# Verify new pods are running
kubectl get pods -n serwaa -l app=avatar-engine -o wide

2.2 Blue-Green Deployment#

bash
# Deploy new version as "green" alongside existing "blue"
helm install serwaa-avatar-engine-green ./charts/avatar-engine \
  --namespace serwaa \
  --set image.tag=v1.2.3 \
  --set service.name=avatar-engine-green \
  --set replicaCount=3

# Run smoke tests against green deployment
kubectl run smoke-test --rm -it --image=curlimages/curl -- \
  curl -s http://avatar-engine-green:8001/health

# Switch traffic (update ingress or service selector)
kubectl patch service avatar-engine -n serwaa \
  -p '{"spec":{"selector":{"version":"green"}}}'

# Verify traffic is flowing to green
kubectl logs -f -l app=avatar-engine,version=green -n serwaa | head -50

# Cleanup blue deployment after verification
helm uninstall serwaa-avatar-engine-blue -n serwaa

2.3 Canary Deployment#

bash
# Deploy canary with 10% traffic
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
  --namespace serwaa \
  --set canary.enabled=true \
  --set canary.weight=10 \
  --set canary.image.tag=v1.2.3

# Monitor canary metrics
kubectl get virtualservice avatar-engine -n serwaa -o yaml

# Gradually increase canary weight
for weight in 25 50 75 100; do
  helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
    --set canary.weight=$weight
  echo "Canary weight: $weight% - waiting 5 minutes..."
  sleep 300
done

3. Rolling Back a Deployment#

bash
# Check rollout history
kubectl rollout history deployment/serwaa-avatar-engine -n serwaa

# Rollback to previous version
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa

# Rollback to specific revision
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa --to-revision=3

# If using Helm
helm rollback serwaa-avatar-engine 1 -n serwaa

4. Model Deployment#

4.1 Promoting a Model to Production#

bash
# Validate model in staging
curl -X POST https://staging.api.serwaa.ai/api/v1/models/validate \
  -H "Authorization: Bearer $API_TOKEN" \
  -d '{"model_name": "3dgs-avatar", "version": "1.3.0"}'

# Promote to production
aws s3 cp s3://serwaa-models-staging/avatar/3dgs/v1.3.0/ \
          s3://serwaa-models/avatar/3dgs/v1.3.0/ --recursive

# Update model registry index
python scripts/update-model-registry.py \
  --model 3dgs-avatar \
  --version 1.3.0 \
  --status production

# Trigger hot-reload in services
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa

4.2 Rolling Back a Model#

bash
# Identify previous version
aws s3 ls s3://serwaa-models/avatar/3dgs/

# Update current version pointer
aws s3 cp s3://serwaa-models/avatar/3dgs/v1.2.0/metadata.json \
          s3://serwaa-models/avatar/3dgs/current/metadata.json

# Restart services to pick up change
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa

# Verify model version in service
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  curl localhost:8001/models/active

GPU Troubleshooting#

1. GPU Not Detected#

Symptoms:

  • Pod stuck in Pending state
  • CUDA initialization errors in logs

Diagnosis:

bash
# Check if GPU nodes are available
kubectl get nodes -l nvidia.com/gpu.present=true

# Check GPU allocatable resources
kubectl describe node <gpu-node> | grep -A10 "Allocatable:"

# Check NVIDIA device plugin status
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds

# Check device plugin logs
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds

Resolution:

bash
# If device plugin is not running
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml

# If node needs driver update
ssh admin@<gpu-node>
sudo apt update && sudo apt install nvidia-driver-535

# Restart kubelet after driver update
sudo systemctl restart kubelet

2. CUDA Out of Memory (OOM)#

Symptoms:

  • CUDA out of memory errors
  • Pod killed with OOMKilled status
  • Service crashes during inference

Diagnosis:

bash
# Check GPU memory usage
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi

# Check pod resource limits
kubectl describe pod <pod-name> -n serwaa | grep -A5 "Limits:"

# Monitor GPU memory over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  watch -n 1 'nvidia-smi --query-gpu=memory.used,memory.total --format=csv'

Resolution:

bash
# 1. Increase GPU memory limits (if available)
kubectl patch deployment serwaa-avatar-engine -n serwaa \
  --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/nvidia.com~1gpu", "value": "2"}]'

# 2. Enable memory optimization in service config
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:512" \
  TORCH_CUDNN_V8_API_ENABLED=1

# 3. Reduce batch size or model precision
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  MODEL_PRECISION=fp16 \
  MAX_BATCH_SIZE=1

# 4. Clear GPU cache manually (within pod)
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; torch.cuda.empty_cache()"

3. GPU Thermal Throttling#

Symptoms:

  • Inference latency gradually increasing
  • GPU clock speed reduced
  • High GPU temperature warnings

Diagnosis:

bash
# Check GPU temperature
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks.current.graphics --format=csv

# Monitor over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  nvidia-smi dmon -s puc -d 5

Resolution:

bash
# 1. Check cooling system on physical nodes
ssh admin@<gpu-node>
sudo ipmitool sensor list | grep -i fan
sudo ipmitool chassis status

# 2. Reduce power limit temporarily
ssh admin@<gpu-node>
sudo nvidia-smi -pl 200  # Reduce from default 350W

# 3. Spread workload across multiple GPUs
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4

4. CUDA Driver Mismatch#

Symptoms:

  • CUDA driver version is insufficient for CUDA runtime version
  • Services fail to start

Diagnosis:

bash
# Check driver version on node
ssh admin@<gpu-node>
nvidia-smi --query-gpu=driver_version --format=csv

# Check CUDA version required by container
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; print(torch.version.cuda)"

Resolution:

bash
# Update NVIDIA driver on node
ssh admin@<gpu-node>
sudo apt update
sudo apt install nvidia-driver-535  # Match container requirements
sudo reboot

# Or use older container image with matching CUDA
kubectl set image deployment/serwaa-avatar-engine -n serwaa \
  avatar-engine=ghcr.io/iron-forge/serwaa/avatar-engine:v1.2.0-cuda11.8

Performance Tuning#

1. Avatar Rendering Optimization#

1.1 Latency Reduction#

bash
# Enable TensorRT for faster inference
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  USE_TENSORRT=true \
  TENSORRT_PRECISION=fp16

# Reduce Gaussian count for faster rendering
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  MAX_GAUSSIANS=75000 \
  SH_DEGREE=2

# Enable CUDA graph optimization
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  ENABLE_CUDA_GRAPHS=true \
  CUDA_GRAPH_WARMUP_ITERATIONS=100

1.2 Throughput Improvement#

bash
# Enable batching
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
  ENABLE_BATCHING=true \
  MAX_BATCH_SIZE=4 \
  BATCH_TIMEOUT_MS=5

# Scale replicas based on GPU availability
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=3

# Configure HPA for auto-scaling
kubectl apply -f - <<EOF
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: avatar-engine-hpa
  namespace: serwaa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: serwaa-avatar-engine
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: nvidia.com/gpu
      target:
        type: Utilization
        averageUtilization: 70
EOF

2. Voice Engine Optimization#

bash
# Enable streaming mode for lower latency
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  TTS_STREAMING_ENABLED=true \
  TTS_CHUNK_SIZE_MS=100

# Use INT8 quantization for Whisper
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  WHISPER_QUANTIZATION=int8 \
  WHISPER_BATCH_SIZE=1

# Enable voice caching
kubectl set env deployment/serwaa-voice-engine -n serwaa \
  VOICE_CACHE_ENABLED=true \
  VOICE_CACHE_MAX_SIZE_MB=1024

3. Database Optimization#

bash
# PostgreSQL performance tuning
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  ALTER SYSTEM SET shared_buffers = '4GB';
  ALTER SYSTEM SET effective_cache_size = '12GB';
  ALTER SYSTEM SET work_mem = '256MB';
  ALTER SYSTEM SET maintenance_work_mem = '1GB';
  ALTER SYSTEM SET random_page_cost = 1.1;
  ALTER SYSTEM SET effective_io_concurrency = 200;
  ALTER SYSTEM SET max_worker_processes = 8;
  ALTER SYSTEM SET max_parallel_workers_per_gather = 4;
  SELECT pg_reload_conf();
"

# Analyze and vacuum tables
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  ANALYZE;
  VACUUM ANALYZE sessions;
  VACUUM ANALYZE personas;
  VACUUM ANALYZE knowledge_documents;
"

# Create missing indexes
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
  CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_status
    ON sessions(status) WHERE status IN ('active', 'pending');
  CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_persona
    ON sessions(persona_id, created_at DESC);
"

4. Redis Optimization#

bash
# Check memory usage
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli INFO memory

# Adjust memory policy
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli CONFIG SET \
  maxmemory 2gb \
  maxmemory-policy allkeys-lru

# Enable compression for large values
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  REDIS_COMPRESSION=lz4 \
  REDIS_COMPRESSION_THRESHOLD_BYTES=1024

5. Network Optimization#

bash
# Enable HTTP/2 for gRPC
kubectl annotate ingress serwaa-ingress -n serwaa \
  nginx.ingress.kubernetes.io/backend-protocol="GRPC"

# Configure connection pooling
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  HTTP_POOL_SIZE=100 \
  HTTP_POOL_TIMEOUT_MS=30000 \
  GRPC_POOL_SIZE=50

Incident Response#

1. Incident Severity Levels#

Level Impact Response Time Examples
P1/SEV1 Complete outage 15 minutes All sessions failing, data loss
P2/SEV2 Significant degradation 1 hour 50%+ latency increase, 10% errors
P3/SEV3 Minor issues 4 hours Single service degraded
P4/SEV4 Minimal impact 24 hours Non-critical bug, cosmetic

2. Incident Response Playbook#

2.1 Initial Assessment (First 5 minutes)#

bash
# 1. Check overall system health
kubectl get pods -n serwaa --field-selector=status.phase!=Running

# 2. Check recent events
kubectl get events -n serwaa --sort-by='.lastTimestamp' | tail -20

# 3. Check service health endpoints
for svc in api-gateway avatar-engine voice-engine; do
  echo "=== $svc ==="
  kubectl exec -it deploy/serwaa-$svc -n serwaa -- curl -s localhost:8000/health
done

# 4. Check error rates in metrics
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..'}[5m])"

# 5. Check recent logs for errors
kubectl logs -l app.kubernetes.io/instance=serwaa -n serwaa --since=5m | grep -i error | tail -50

2.2 Communication Template#

text
INCIDENT DECLARED: [P1/P2/P3/P4]

Time: [UTC timestamp]
Impact: [Brief description of user impact]
Affected Services: [List of services]
Status: [Investigating/Identified/Mitigated/Resolved]

Current Actions:
- [Action 1]
- [Action 2]

Next Update: [Time of next update]

Incident Commander: [Name]
Communications: [#serwaa-incidents Slack channel]

2.3 P1 Response: Complete Outage#

bash
# Step 1: Identify failing components
kubectl get pods -n serwaa -o wide
kubectl get svc -n serwaa

# Step 2: Check for resource exhaustion
kubectl top pods -n serwaa
kubectl top nodes

# Step 3: Check external dependencies
# Database
kubectl exec -it serwaa-postgresql-0 -n serwaa -- pg_isready
# Redis
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli ping
# S3
aws s3 ls s3://serwaa-models/ --max-items 1

# Step 4: If database is down
kubectl rollout restart statefulset/serwaa-postgresql -n serwaa

# Step 5: If application pods are crashing
kubectl describe pod <crashing-pod> -n serwaa
kubectl logs <crashing-pod> -n serwaa --previous

# Step 6: Emergency rollback if recent deployment
helm rollback serwaa-api-gateway -n serwaa
helm rollback serwaa-avatar-engine -n serwaa

# Step 7: Scale up for capacity issues
kubectl scale deployment/serwaa-api-gateway -n serwaa --replicas=5

2.4 P2 Response: Performance Degradation#

bash
# Step 1: Identify slow services
kubectl top pods -n serwaa --containers | sort -k4 -rn | head -10

# Step 2: Check latency metrics
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(http_request_duration_seconds_bucket[5m]))"

# Step 3: Check for hotspots
kubectl exec -it deploy/serwaa-api-gateway -n serwaa -- \
  cat /tmp/profiling/hotspots.log

# Step 4: Check GPU utilization
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi

# Step 5: Increase capacity
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4

# Step 6: Enable rate limiting to protect system
kubectl set env deployment/serwaa-api-gateway -n serwaa \
  RATE_LIMIT_ENABLED=true \
  RATE_LIMIT_REQUESTS_PER_MINUTE=50

3. Post-Incident Actions#

bash
# 1. Collect diagnostic data
kubectl cluster-info dump --output-directory=/tmp/incident-$(date +%Y%m%d-%H%M%S)

# 2. Export relevant logs
kubectl logs -l app=serwaa -n serwaa --since=2h > incident-logs.txt

# 3. Export metrics
curl -s "http://prometheus:9090/api/v1/query_range?query=up&start=$(date -d '2 hours ago' +%s)&end=$(date +%s)&step=60" > metrics-export.json

# 4. Generate incident report
cat > incident-report.md <<EOF
# Incident Report: [INCIDENT-YYYY-MM-DD-XXX]

## Summary
[Brief description]

## Timeline
- [HH:MM UTC] - Incident detected
- [HH:MM UTC] - Incident declared
- [HH:MM UTC] - Root cause identified
- [HH:MM UTC] - Mitigation applied
- [HH:MM UTC] - Incident resolved

## Root Cause
[Detailed explanation]

## Impact
- Duration: X hours Y minutes
- Affected users: N
- Failed requests: N

## Action Items
- [ ] [Action 1]
- [ ] [Action 2]

## Lessons Learned
[What we learned]
EOF

Common Operations#

1. Scaling Services#

bash
# Manual scaling
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=5

# Check current replicas
kubectl get deployment -n serwaa -o wide

# Configure HPA
kubectl autoscale deployment/serwaa-avatar-engine -n serwaa \
  --min=2 --max=10 --cpu-percent=70

2. Viewing Logs#

bash
# Stream logs from all pods of a service
kubectl logs -f -l app=avatar-engine -n serwaa --all-containers

# Search for specific patterns
kubectl logs -l app=api-gateway -n serwaa --since=1h | grep -i "error\|exception"

# Export logs to file
kubectl logs deploy/serwaa-api-gateway -n serwaa --since=24h > api-gateway-logs.txt

# Tail last N lines
kubectl logs deploy/serwaa-voice-engine -n serwaa --tail=100

3. Debugging Pods#

bash
# Get shell access
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- /bin/bash

# Run diagnostics inside pod
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
  python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0)}')"

# Copy files from pod
kubectl cp serwaa/serwaa-avatar-engine-xxx:/tmp/debug.log ./debug.log

# Port-forward for local access
kubectl port-forward deploy/serwaa-api-gateway -n serwaa 8000:8000

4. Certificate Management#

bash
# Check certificate expiry
kubectl get certificate -n serwaa
kubectl describe certificate serwaa-tls -n serwaa

# Manually trigger renewal
kubectl delete certificate serwaa-tls -n serwaa
kubectl apply -f certificates/serwaa-tls.yaml

# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager

5. Secret Rotation#

bash
# Rotate database password
NEW_PASSWORD=$(openssl rand -base64 32)
kubectl create secret generic db-credentials \
  --from-literal=password=$NEW_PASSWORD \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart services to pick up new secrets
kubectl rollout restart deployment -n serwaa

# Rotate API keys
kubectl patch secret api-keys -n serwaa \
  --patch="{\"stringData\":{\"anthropic-api-key\":\"$NEW_API_KEY\"}}"

Monitoring & Alerting#

1. Key Metrics to Monitor#

Metric Warning Threshold Critical Threshold
API Gateway Error Rate > 1% > 5%
Avatar Render Latency (p99) > 50ms > 100ms
Voice Synthesis Latency (p99) > 150ms > 300ms
GPU Memory Utilization > 80% > 95%
GPU Temperature > 75°C > 85°C
Active Sessions > 80% capacity > 95% capacity
Database Connections > 80% max > 95% max
Redis Memory > 80% max > 95% max

2. Alert Configuration#

yaml
# prometheus-alerts.yaml
groups:
  - name: serwaa-alerts
    rules:
      - alert: HighErrorRate
        expr:
          rate(http_requests_total{status=~"5.."}[5m]) /
          rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: 'High error rate in {{ $labels.service }}'

      - alert: HighLatency
        expr:
          histogram_quantile(0.99,
          rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High latency in {{ $labels.service }}'

      - alert: GPUMemoryHigh
        expr:
          nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes > 0.95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'GPU memory exhausted on {{ $labels.instance }}'

      - alert: GPUTemperatureHigh
        expr: nvidia_gpu_temperature_celsius > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'GPU overheating on {{ $labels.instance }}'

3. Grafana Dashboards#

Access Grafana at https://grafana.serwaa.ai (or port-forward locally):

bash
kubectl port-forward svc/grafana -n monitoring 3000:80

Key Dashboards:

  • Serwaa Overview: High-level system health
  • Avatar Engine: GPU metrics, render latency, FPS
  • Voice Engine: TTS/STT latency, success rates
  • API Gateway: Request rates, error rates, latencies
  • Database: Connection pools, query performance
  • Redis: Memory usage, hit rates, latency

4. Log Aggregation#

bash
# Access Kibana
kubectl port-forward svc/kibana -n logging 5601:5601

# Query logs via CLI
kubectl exec -it deploy/elasticsearch -n logging -- \
  curl -s "localhost:9200/serwaa-*/_search?q=level:error&size=10" | jq

# Create log-based alerts
curl -X PUT "http://elasticsearch:9200/_watcher/watch/high-error-rate" \
  -H "Content-Type: application/json" \
  -d @error-rate-watch.json

Emergency Contacts#

Role Contact Escalation Time
On-Call Engineer PagerDuty: serwaa-eng Immediate
Platform Lead [Email/Phone] 15 minutes
Engineering Manager [Email/Phone] 30 minutes
VP Engineering [Email/Phone] 1 hour (P1 only)


Revision History#

Version Date Author Changes
1.0.0 2026-01-31 Claude Opus Initial runbooks