This document provides operational runbooks for the Psyche (Serwaa) AI Virtual Assistant Platform, covering deployment procedures, GPU troubleshooting, performance tuning, and incident response.
Table of Contents#
- Deployment Procedures
- GPU Troubleshooting
- Performance Tuning
- Incident Response
- Common Operations
- Monitoring & Alerting
Deployment Procedures#
1. Pre-Deployment Checklist#
Before deploying any Psyche service:
bash
# 1. Verify cluster health
kubectl get nodes -o wide
kubectl get pods -n serwaa --field-selector=status.phase!=Running
# 2. Check GPU availability
kubectl describe nodes | grep -A5 "nvidia.com/gpu"
# 3. Verify secrets exist
kubectl get secrets -n serwaa | grep -E "(db-credentials|api-keys|tls)"
# 4. Check persistent volumes
kubectl get pv,pvc -n serwaa
# 5. Verify model registry access
aws s3 ls s3://serwaa-models/registry/index.json
2. Deploying a New Service Version#
2.1 Rolling Deployment (Default)#
bash
# Update image tag in values.yaml or use --set
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
--namespace serwaa \
--set image.tag=v1.2.3 \
--set image.pullPolicy=Always \
--wait \
--timeout 10m
# Monitor rollout
kubectl rollout status deployment/serwaa-avatar-engine -n serwaa
# Verify new pods are running
kubectl get pods -n serwaa -l app=avatar-engine -o wide
2.2 Blue-Green Deployment#
bash
# Deploy new version as "green" alongside existing "blue"
helm install serwaa-avatar-engine-green ./charts/avatar-engine \
--namespace serwaa \
--set image.tag=v1.2.3 \
--set service.name=avatar-engine-green \
--set replicaCount=3
# Run smoke tests against green deployment
kubectl run smoke-test --rm -it --image=curlimages/curl -- \
curl -s http://avatar-engine-green:8001/health
# Switch traffic (update ingress or service selector)
kubectl patch service avatar-engine -n serwaa \
-p '{"spec":{"selector":{"version":"green"}}}'
# Verify traffic is flowing to green
kubectl logs -f -l app=avatar-engine,version=green -n serwaa | head -50
# Cleanup blue deployment after verification
helm uninstall serwaa-avatar-engine-blue -n serwaa
2.3 Canary Deployment#
bash
# Deploy canary with 10% traffic
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
--namespace serwaa \
--set canary.enabled=true \
--set canary.weight=10 \
--set canary.image.tag=v1.2.3
# Monitor canary metrics
kubectl get virtualservice avatar-engine -n serwaa -o yaml
# Gradually increase canary weight
for weight in 25 50 75 100; do
helm upgrade serwaa-avatar-engine ./charts/avatar-engine \
--set canary.weight=$weight
echo "Canary weight: $weight% - waiting 5 minutes..."
sleep 300
done
3. Rolling Back a Deployment#
bash
# Check rollout history
kubectl rollout history deployment/serwaa-avatar-engine -n serwaa
# Rollback to previous version
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa
# Rollback to specific revision
kubectl rollout undo deployment/serwaa-avatar-engine -n serwaa --to-revision=3
# If using Helm
helm rollback serwaa-avatar-engine 1 -n serwaa
4. Model Deployment#
4.1 Promoting a Model to Production#
bash
# Validate model in staging
curl -X POST https://staging.api.serwaa.ai/api/v1/models/validate \
-H "Authorization: Bearer $API_TOKEN" \
-d '{"model_name": "3dgs-avatar", "version": "1.3.0"}'
# Promote to production
aws s3 cp s3://serwaa-models-staging/avatar/3dgs/v1.3.0/ \
s3://serwaa-models/avatar/3dgs/v1.3.0/ --recursive
# Update model registry index
python scripts/update-model-registry.py \
--model 3dgs-avatar \
--version 1.3.0 \
--status production
# Trigger hot-reload in services
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa
4.2 Rolling Back a Model#
bash
# Identify previous version
aws s3 ls s3://serwaa-models/avatar/3dgs/
# Update current version pointer
aws s3 cp s3://serwaa-models/avatar/3dgs/v1.2.0/metadata.json \
s3://serwaa-models/avatar/3dgs/current/metadata.json
# Restart services to pick up change
kubectl rollout restart deployment/serwaa-avatar-engine -n serwaa
# Verify model version in service
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
curl localhost:8001/models/active
GPU Troubleshooting#
1. GPU Not Detected#
Symptoms:
- Pod stuck in Pending state
- CUDA initialization errors in logs
Diagnosis:
bash
# Check if GPU nodes are available
kubectl get nodes -l nvidia.com/gpu.present=true
# Check GPU allocatable resources
kubectl describe node <gpu-node> | grep -A10 "Allocatable:"
# Check NVIDIA device plugin status
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
# Check device plugin logs
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds
Resolution:
bash
# If device plugin is not running
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml
# If node needs driver update
ssh admin@<gpu-node>
sudo apt update && sudo apt install nvidia-driver-535
# Restart kubelet after driver update
sudo systemctl restart kubelet
2. CUDA Out of Memory (OOM)#
Symptoms:
CUDA out of memoryerrors- Pod killed with OOMKilled status
- Service crashes during inference
Diagnosis:
bash
# Check GPU memory usage
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi
# Check pod resource limits
kubectl describe pod <pod-name> -n serwaa | grep -A5 "Limits:"
# Monitor GPU memory over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
watch -n 1 'nvidia-smi --query-gpu=memory.used,memory.total --format=csv'
Resolution:
bash
# 1. Increase GPU memory limits (if available)
kubectl patch deployment serwaa-avatar-engine -n serwaa \
--type=json \
-p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/nvidia.com~1gpu", "value": "2"}]'
# 2. Enable memory optimization in service config
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:512" \
TORCH_CUDNN_V8_API_ENABLED=1
# 3. Reduce batch size or model precision
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
MODEL_PRECISION=fp16 \
MAX_BATCH_SIZE=1
# 4. Clear GPU cache manually (within pod)
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
python -c "import torch; torch.cuda.empty_cache()"
3. GPU Thermal Throttling#
Symptoms:
- Inference latency gradually increasing
- GPU clock speed reduced
- High GPU temperature warnings
Diagnosis:
bash
# Check GPU temperature
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks.current.graphics --format=csv
# Monitor over time
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
nvidia-smi dmon -s puc -d 5
Resolution:
bash
# 1. Check cooling system on physical nodes
ssh admin@<gpu-node>
sudo ipmitool sensor list | grep -i fan
sudo ipmitool chassis status
# 2. Reduce power limit temporarily
ssh admin@<gpu-node>
sudo nvidia-smi -pl 200 # Reduce from default 350W
# 3. Spread workload across multiple GPUs
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4
4. CUDA Driver Mismatch#
Symptoms:
CUDA driver version is insufficient for CUDA runtime version- Services fail to start
Diagnosis:
bash
# Check driver version on node
ssh admin@<gpu-node>
nvidia-smi --query-gpu=driver_version --format=csv
# Check CUDA version required by container
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
python -c "import torch; print(torch.version.cuda)"
Resolution:
bash
# Update NVIDIA driver on node
ssh admin@<gpu-node>
sudo apt update
sudo apt install nvidia-driver-535 # Match container requirements
sudo reboot
# Or use older container image with matching CUDA
kubectl set image deployment/serwaa-avatar-engine -n serwaa \
avatar-engine=ghcr.io/iron-forge/serwaa/avatar-engine:v1.2.0-cuda11.8
Performance Tuning#
1. Avatar Rendering Optimization#
1.1 Latency Reduction#
bash
# Enable TensorRT for faster inference
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
USE_TENSORRT=true \
TENSORRT_PRECISION=fp16
# Reduce Gaussian count for faster rendering
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
MAX_GAUSSIANS=75000 \
SH_DEGREE=2
# Enable CUDA graph optimization
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
ENABLE_CUDA_GRAPHS=true \
CUDA_GRAPH_WARMUP_ITERATIONS=100
1.2 Throughput Improvement#
bash
# Enable batching
kubectl set env deployment/serwaa-avatar-engine -n serwaa \
ENABLE_BATCHING=true \
MAX_BATCH_SIZE=4 \
BATCH_TIMEOUT_MS=5
# Scale replicas based on GPU availability
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=3
# Configure HPA for auto-scaling
kubectl apply -f - <<EOF
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: avatar-engine-hpa
namespace: serwaa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: serwaa-avatar-engine
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: nvidia.com/gpu
target:
type: Utilization
averageUtilization: 70
EOF
2. Voice Engine Optimization#
bash
# Enable streaming mode for lower latency
kubectl set env deployment/serwaa-voice-engine -n serwaa \
TTS_STREAMING_ENABLED=true \
TTS_CHUNK_SIZE_MS=100
# Use INT8 quantization for Whisper
kubectl set env deployment/serwaa-voice-engine -n serwaa \
WHISPER_QUANTIZATION=int8 \
WHISPER_BATCH_SIZE=1
# Enable voice caching
kubectl set env deployment/serwaa-voice-engine -n serwaa \
VOICE_CACHE_ENABLED=true \
VOICE_CACHE_MAX_SIZE_MB=1024
3. Database Optimization#
bash
# PostgreSQL performance tuning
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
ALTER SYSTEM SET shared_buffers = '4GB';
ALTER SYSTEM SET effective_cache_size = '12GB';
ALTER SYSTEM SET work_mem = '256MB';
ALTER SYSTEM SET maintenance_work_mem = '1GB';
ALTER SYSTEM SET random_page_cost = 1.1;
ALTER SYSTEM SET effective_io_concurrency = 200;
ALTER SYSTEM SET max_worker_processes = 8;
ALTER SYSTEM SET max_parallel_workers_per_gather = 4;
SELECT pg_reload_conf();
"
# Analyze and vacuum tables
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
ANALYZE;
VACUUM ANALYZE sessions;
VACUUM ANALYZE personas;
VACUUM ANALYZE knowledge_documents;
"
# Create missing indexes
kubectl exec -it serwaa-postgresql-0 -n serwaa -- psql -U serwaa -c "
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_status
ON sessions(status) WHERE status IN ('active', 'pending');
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_sessions_persona
ON sessions(persona_id, created_at DESC);
"
4. Redis Optimization#
bash
# Check memory usage
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli INFO memory
# Adjust memory policy
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli CONFIG SET \
maxmemory 2gb \
maxmemory-policy allkeys-lru
# Enable compression for large values
kubectl set env deployment/serwaa-api-gateway -n serwaa \
REDIS_COMPRESSION=lz4 \
REDIS_COMPRESSION_THRESHOLD_BYTES=1024
5. Network Optimization#
bash
# Enable HTTP/2 for gRPC
kubectl annotate ingress serwaa-ingress -n serwaa \
nginx.ingress.kubernetes.io/backend-protocol="GRPC"
# Configure connection pooling
kubectl set env deployment/serwaa-api-gateway -n serwaa \
HTTP_POOL_SIZE=100 \
HTTP_POOL_TIMEOUT_MS=30000 \
GRPC_POOL_SIZE=50
Incident Response#
1. Incident Severity Levels#
| Level | Impact | Response Time | Examples |
|---|---|---|---|
| P1/SEV1 | Complete outage | 15 minutes | All sessions failing, data loss |
| P2/SEV2 | Significant degradation | 1 hour | 50%+ latency increase, 10% errors |
| P3/SEV3 | Minor issues | 4 hours | Single service degraded |
| P4/SEV4 | Minimal impact | 24 hours | Non-critical bug, cosmetic |
2. Incident Response Playbook#
2.1 Initial Assessment (First 5 minutes)#
bash
# 1. Check overall system health
kubectl get pods -n serwaa --field-selector=status.phase!=Running
# 2. Check recent events
kubectl get events -n serwaa --sort-by='.lastTimestamp' | tail -20
# 3. Check service health endpoints
for svc in api-gateway avatar-engine voice-engine; do
echo "=== $svc ==="
kubectl exec -it deploy/serwaa-$svc -n serwaa -- curl -s localhost:8000/health
done
# 4. Check error rates in metrics
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..'}[5m])"
# 5. Check recent logs for errors
kubectl logs -l app.kubernetes.io/instance=serwaa -n serwaa --since=5m | grep -i error | tail -50
2.2 Communication Template#
text
INCIDENT DECLARED: [P1/P2/P3/P4]
Time: [UTC timestamp]
Impact: [Brief description of user impact]
Affected Services: [List of services]
Status: [Investigating/Identified/Mitigated/Resolved]
Current Actions:
- [Action 1]
- [Action 2]
Next Update: [Time of next update]
Incident Commander: [Name]
Communications: [#serwaa-incidents Slack channel]
2.3 P1 Response: Complete Outage#
bash
# Step 1: Identify failing components
kubectl get pods -n serwaa -o wide
kubectl get svc -n serwaa
# Step 2: Check for resource exhaustion
kubectl top pods -n serwaa
kubectl top nodes
# Step 3: Check external dependencies
# Database
kubectl exec -it serwaa-postgresql-0 -n serwaa -- pg_isready
# Redis
kubectl exec -it serwaa-redis-master-0 -n serwaa -- redis-cli ping
# S3
aws s3 ls s3://serwaa-models/ --max-items 1
# Step 4: If database is down
kubectl rollout restart statefulset/serwaa-postgresql -n serwaa
# Step 5: If application pods are crashing
kubectl describe pod <crashing-pod> -n serwaa
kubectl logs <crashing-pod> -n serwaa --previous
# Step 6: Emergency rollback if recent deployment
helm rollback serwaa-api-gateway -n serwaa
helm rollback serwaa-avatar-engine -n serwaa
# Step 7: Scale up for capacity issues
kubectl scale deployment/serwaa-api-gateway -n serwaa --replicas=5
2.4 P2 Response: Performance Degradation#
bash
# Step 1: Identify slow services
kubectl top pods -n serwaa --containers | sort -k4 -rn | head -10
# Step 2: Check latency metrics
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(http_request_duration_seconds_bucket[5m]))"
# Step 3: Check for hotspots
kubectl exec -it deploy/serwaa-api-gateway -n serwaa -- \
cat /tmp/profiling/hotspots.log
# Step 4: Check GPU utilization
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- nvidia-smi
# Step 5: Increase capacity
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=4
# Step 6: Enable rate limiting to protect system
kubectl set env deployment/serwaa-api-gateway -n serwaa \
RATE_LIMIT_ENABLED=true \
RATE_LIMIT_REQUESTS_PER_MINUTE=50
3. Post-Incident Actions#
bash
# 1. Collect diagnostic data
kubectl cluster-info dump --output-directory=/tmp/incident-$(date +%Y%m%d-%H%M%S)
# 2. Export relevant logs
kubectl logs -l app=serwaa -n serwaa --since=2h > incident-logs.txt
# 3. Export metrics
curl -s "http://prometheus:9090/api/v1/query_range?query=up&start=$(date -d '2 hours ago' +%s)&end=$(date +%s)&step=60" > metrics-export.json
# 4. Generate incident report
cat > incident-report.md <<EOF
# Incident Report: [INCIDENT-YYYY-MM-DD-XXX]
## Summary
[Brief description]
## Timeline
- [HH:MM UTC] - Incident detected
- [HH:MM UTC] - Incident declared
- [HH:MM UTC] - Root cause identified
- [HH:MM UTC] - Mitigation applied
- [HH:MM UTC] - Incident resolved
## Root Cause
[Detailed explanation]
## Impact
- Duration: X hours Y minutes
- Affected users: N
- Failed requests: N
## Action Items
- [ ] [Action 1]
- [ ] [Action 2]
## Lessons Learned
[What we learned]
EOF
Common Operations#
1. Scaling Services#
bash
# Manual scaling
kubectl scale deployment/serwaa-avatar-engine -n serwaa --replicas=5
# Check current replicas
kubectl get deployment -n serwaa -o wide
# Configure HPA
kubectl autoscale deployment/serwaa-avatar-engine -n serwaa \
--min=2 --max=10 --cpu-percent=70
2. Viewing Logs#
bash
# Stream logs from all pods of a service
kubectl logs -f -l app=avatar-engine -n serwaa --all-containers
# Search for specific patterns
kubectl logs -l app=api-gateway -n serwaa --since=1h | grep -i "error\|exception"
# Export logs to file
kubectl logs deploy/serwaa-api-gateway -n serwaa --since=24h > api-gateway-logs.txt
# Tail last N lines
kubectl logs deploy/serwaa-voice-engine -n serwaa --tail=100
3. Debugging Pods#
bash
# Get shell access
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- /bin/bash
# Run diagnostics inside pod
kubectl exec -it deploy/serwaa-avatar-engine -n serwaa -- \
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, Device: {torch.cuda.get_device_name(0)}')"
# Copy files from pod
kubectl cp serwaa/serwaa-avatar-engine-xxx:/tmp/debug.log ./debug.log
# Port-forward for local access
kubectl port-forward deploy/serwaa-api-gateway -n serwaa 8000:8000
4. Certificate Management#
bash
# Check certificate expiry
kubectl get certificate -n serwaa
kubectl describe certificate serwaa-tls -n serwaa
# Manually trigger renewal
kubectl delete certificate serwaa-tls -n serwaa
kubectl apply -f certificates/serwaa-tls.yaml
# Check cert-manager logs
kubectl logs -n cert-manager -l app=cert-manager
5. Secret Rotation#
bash
# Rotate database password
NEW_PASSWORD=$(openssl rand -base64 32)
kubectl create secret generic db-credentials \
--from-literal=password=$NEW_PASSWORD \
--dry-run=client -o yaml | kubectl apply -f -
# Restart services to pick up new secrets
kubectl rollout restart deployment -n serwaa
# Rotate API keys
kubectl patch secret api-keys -n serwaa \
--patch="{\"stringData\":{\"anthropic-api-key\":\"$NEW_API_KEY\"}}"
Monitoring & Alerting#
1. Key Metrics to Monitor#
| Metric | Warning Threshold | Critical Threshold |
|---|---|---|
| API Gateway Error Rate | > 1% | > 5% |
| Avatar Render Latency (p99) | > 50ms | > 100ms |
| Voice Synthesis Latency (p99) | > 150ms | > 300ms |
| GPU Memory Utilization | > 80% | > 95% |
| GPU Temperature | > 75°C | > 85°C |
| Active Sessions | > 80% capacity | > 95% capacity |
| Database Connections | > 80% max | > 95% max |
| Redis Memory | > 80% max | > 95% max |
2. Alert Configuration#
yaml
# prometheus-alerts.yaml
groups:
- name: serwaa-alerts
rules:
- alert: HighErrorRate
expr:
rate(http_requests_total{status=~"5.."}[5m]) /
rate(http_requests_total[5m]) > 0.05
for: 2m
labels:
severity: critical
annotations:
summary: 'High error rate in {{ $labels.service }}'
- alert: HighLatency
expr:
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: 'High latency in {{ $labels.service }}'
- alert: GPUMemoryHigh
expr:
nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes > 0.95
for: 5m
labels:
severity: critical
annotations:
summary: 'GPU memory exhausted on {{ $labels.instance }}'
- alert: GPUTemperatureHigh
expr: nvidia_gpu_temperature_celsius > 85
for: 5m
labels:
severity: critical
annotations:
summary: 'GPU overheating on {{ $labels.instance }}'
3. Grafana Dashboards#
Access Grafana at https://grafana.serwaa.ai (or port-forward locally):
bash
kubectl port-forward svc/grafana -n monitoring 3000:80
Key Dashboards:
- Serwaa Overview: High-level system health
- Avatar Engine: GPU metrics, render latency, FPS
- Voice Engine: TTS/STT latency, success rates
- API Gateway: Request rates, error rates, latencies
- Database: Connection pools, query performance
- Redis: Memory usage, hit rates, latency
4. Log Aggregation#
bash
# Access Kibana
kubectl port-forward svc/kibana -n logging 5601:5601
# Query logs via CLI
kubectl exec -it deploy/elasticsearch -n logging -- \
curl -s "localhost:9200/serwaa-*/_search?q=level:error&size=10" | jq
# Create log-based alerts
curl -X PUT "http://elasticsearch:9200/_watcher/watch/high-error-rate" \
-H "Content-Type: application/json" \
-d @error-rate-watch.json
Emergency Contacts#
| Role | Contact | Escalation Time |
|---|---|---|
| On-Call Engineer | PagerDuty: serwaa-eng | Immediate |
| Platform Lead | [Email/Phone] | 15 minutes |
| Engineering Manager | [Email/Phone] | 30 minutes |
| VP Engineering | [Email/Phone] | 1 hour (P1 only) |
Related Documentation#
Revision History#
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0.0 | 2026-01-31 | Claude Opus | Initial runbooks |