Operational procedures for scaling Lilith platform services.
Table of Contents#
- Overview
- When to Scale
- Pre-Scaling Checklist
- Manual Scaling
- Autoscaling Configuration
- Service-Specific Scaling
- Database Scaling
- Cache Scaling
- Post-Scaling Validation
- Troubleshooting
Overview#
The Lilith platform uses Kubernetes Horizontal Pod Autoscaler (HPA) for automatic scaling and supports manual scaling for planned capacity changes.
Architecture Components#
| Component | Scaling Method | Min Replicas | Max Replicas |
|---|---|---|---|
| BFF | HPA (CPU/Memory) | 3 | 20 |
| Auth | HPA (CPU) | 2 | 10 |
| Content | HPA (CPU) | 2 | 15 |
| AI | HPA (Custom: Queue Depth) | 3 | 30 |
| Conversation | HPA (Connections) | 2 | 20 |
| Media | HPA (CPU) | 2 | 10 |
| Notification | HPA (Queue Depth) | 2 | 15 |
| PostgreSQL | Manual (Vertical) | 1 primary | 3 replicas |
| Redis | Manual (Cluster) | 3 nodes | 9 nodes |
When to Scale#
Scale UP Indicators#
| Metric | Threshold | Action |
|---|---|---|
| CPU utilization | > 70% sustained 5min | Scale up |
| Memory utilization | > 80% sustained 5min | Scale up |
| Request latency p95 | > 500ms sustained 5min | Scale up |
| Error rate | > 1% sustained 2min | Investigate, possibly scale |
| Queue depth | > 1000 messages sustained | Scale consumers |
| Connection saturation | > 80% max connections | Scale up |
Scale DOWN Indicators#
| Metric | Threshold | Action |
|---|---|---|
| CPU utilization | < 30% sustained 15min | Scale down |
| Memory utilization | < 40% sustained 15min | Scale down |
| Request latency p95 | < 100ms sustained 15min | Consider scale down |
| Queue depth | < 100 messages sustained | Scale down consumers |
Planned Scaling Events#
Scale proactively for:
- Marketing campaigns (24h before)
- Feature launches (12h before)
- Peak usage times (daily patterns)
- Load testing (before tests)
Pre-Scaling Checklist#
Before scaling any component:
- Verify current resource utilization
- Check cluster capacity (nodes available)
- Verify no pending deployments
- Check database connection pool limits
- Review recent incidents
- Confirm monitoring is active
- Notify on-call if major change
bash
# Check cluster capacity
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory
# Check current pod distribution
kubectl get pods -n production -o wide | grep -v Completed
# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending
Manual Scaling#
Scale Deployment Replicas#
bash
# Scale specific service
kubectl scale deployment auth -n production --replicas=5
# Scale with rollout status
kubectl scale deployment auth -n production --replicas=5 && \
kubectl rollout status deployment/auth -n production
# Verify scaling
kubectl get deployment auth -n production
Scale Multiple Services#
bash
# Scale all API services
for svc in bff auth content conversation; do
kubectl scale deployment $svc -n production --replicas=5
echo "Scaled $svc to 5 replicas"
done
Emergency Scaling Script#
bash
#!/bin/bash
# scripts/emergency-scale.sh
NAMESPACE="${1:-production}"
MULTIPLIER="${2:-2}"
echo "Emergency scaling in $NAMESPACE with ${MULTIPLIER}x multiplier"
# Get current replicas and scale
kubectl get deployments -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.replicas}{"\n"}{end}' | \
while read name replicas; do
new_replicas=$((replicas * MULTIPLIER))
echo "Scaling $name from $replicas to $new_replicas"
kubectl scale deployment $name -n $NAMESPACE --replicas=$new_replicas
done
# Wait for rollout
kubectl rollout status deployment -n $NAMESPACE --timeout=300s
Autoscaling Configuration#
HPA Configuration#
yaml
# Example HPA for BFF service
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: bff-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: bff
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 4
periodSeconds: 60
- type: Percent
value: 100
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Pods
value: 1
periodSeconds: 120
View HPA Status#
bash
# List all HPAs
kubectl get hpa -n production
# Detailed HPA status
kubectl describe hpa bff-hpa -n production
# Watch HPA activity
kubectl get hpa -n production -w
Adjust HPA Limits#
bash
# Update max replicas
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"maxReplicas":30}}'
# Update target CPU
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"metrics":[{"type":"Resource","resource":{"name":"cpu","target":{"type":"Utilization","averageUtilization":60}}}]}}'
Custom Metrics Scaling (AI Service)#
yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ai-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ai
minReplicas: 3
maxReplicas: 30
metrics:
- type: External
external:
metric:
name: rabbitmq_queue_messages
selector:
matchLabels:
queue: ai-requests
target:
type: AverageValue
averageValue: '100'
Service-Specific Scaling#
BFF (Backend-for-Frontend)#
Scaling triggers:
- Aggregates multiple backend calls
- CPU-bound during peak traffic
- Memory usage for response caching
bash
# Scale BFF
kubectl scale deployment bff -n production --replicas=10
# Verify load balancing
for i in {1..10}; do
curl -s http://bff.lilith.app/health | jq '.hostname'
done
Auth Service#
Scaling triggers:
- Login surge during peak hours
- JWT verification CPU load
- Session store operations
Considerations:
- Session affinity not required (stateless JWT)
- Redis connection pool per replica
bash
# Scale auth with connection pool check
CURRENT=$(kubectl get deployment auth -n production -o jsonpath='{.spec.replicas}')
NEW_REPLICAS=10
MAX_CONNECTIONS=100 # Redis max connections
if [ $((NEW_REPLICAS * 10)) -gt $MAX_CONNECTIONS ]; then
echo "Warning: May exceed Redis connection limit"
fi
kubectl scale deployment auth -n production --replicas=$NEW_REPLICAS
AI Service#
Scaling triggers:
- Queue depth > 500 messages
- Response latency > 2s
- GPU utilization > 80%
Considerations:
- GPU pods have longer startup (30-60s)
- Pre-warm during expected traffic surge
- Scale in smaller increments
bash
# Check AI queue depth
kubectl exec -n production deploy/rabbitmq -- rabbitmqctl list_queues | grep ai
# Scale AI with extended timeout
kubectl scale deployment ai -n production --replicas=15
kubectl rollout status deployment/ai -n production --timeout=600s
Conversation Service#
Scaling triggers:
- Active WebSocket connections
- Message throughput
- SSE stream count
Considerations:
- WebSocket connections are stateful
- Use pod disruption budget
- Scale gradually to redistribute connections
bash
# Check active connections per pod
kubectl exec -n production deploy/conversation -- curl -s localhost:3001/metrics | grep websocket_connections
# Scale with PDB protection
kubectl scale deployment conversation -n production --replicas=8
Database Scaling#
PostgreSQL Vertical Scaling#
bash
# 1. Schedule maintenance window
# 2. Update resource requests in values.yaml
# 3. Apply changes
helm upgrade postgresql bitnami/postgresql \
--namespace production \
--set primary.resources.requests.cpu=4 \
--set primary.resources.requests.memory=16Gi \
--set primary.resources.limits.cpu=8 \
--set primary.resources.limits.memory=32Gi
PostgreSQL Read Replicas#
bash
# Add read replica
helm upgrade postgresql bitnami/postgresql \
--namespace production \
--set readReplicas.replicaCount=3
# Verify replication
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT * FROM pg_stat_replication"
Connection Pool Scaling#
yaml
# PgBouncer configuration
pgbouncer:
poolMode: transaction
maxClientConn: 1000
defaultPoolSize: 25
minPoolSize: 5
reservePoolSize: 5
Cache Scaling#
Redis Scaling#
bash
# Add Redis replica
kubectl scale statefulset redis -n production --replicas=5
# Verify cluster
kubectl exec -n production redis-0 -- redis-cli cluster info
# Check memory usage across nodes
for pod in $(kubectl get pods -n production -l app=redis -o name); do
echo "=== $pod ==="
kubectl exec -n production $pod -- redis-cli info memory | grep used_memory_human
done
Redis Cluster Rebalancing#
bash
# After adding nodes, rebalance slots
kubectl exec -n production redis-0 -- redis-cli --cluster rebalance redis-0:6379
Post-Scaling Validation#
Verification Checklist#
bash
#!/bin/bash
# scripts/verify-scaling.sh
NAMESPACE="${1:-production}"
echo "=== Scaling Verification ==="
# 1. Check all pods running
echo "Checking pod status..."
kubectl get pods -n $NAMESPACE | grep -v Running | grep -v Completed
if [ $? -eq 0 ]; then
echo "WARNING: Not all pods are running"
fi
# 2. Check HPA status
echo "Checking HPA status..."
kubectl get hpa -n $NAMESPACE
# 3. Health checks
echo "Checking service health..."
for svc in bff auth content ai; do
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://$svc.$NAMESPACE.svc.cluster.local/health)
echo "$svc: $STATUS"
done
# 4. Latency check
echo "Checking latencies..."
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,http_request_duration_seconds_bucket{namespace='$NAMESPACE'})" | jq '.data.result[] | {service: .metric.service, p95: .value[1]}'
# 5. Error rate check
echo "Checking error rates..."
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..',namespace='$NAMESPACE'}[5m])" | jq '.data.result[] | {service: .metric.service, errors: .value[1]}'
Load Testing After Scale#
bash
# Quick smoke test
for i in {1..100}; do
curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://bff.lilith.app/health
done | sort | uniq -c
# Full load test
k6 run --vus 100 --duration 5m scripts/load-test.js
Troubleshooting#
Pods Not Scaling Up#
bash
# Check HPA events
kubectl describe hpa bff-hpa -n production | tail -20
# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending
# Check node capacity
kubectl describe nodes | grep -A5 "Allocated resources"
# Check for resource quotas
kubectl describe resourcequota -n production
Common causes:
- Insufficient cluster capacity (add nodes)
- Resource quota exceeded
- Pod affinity/anti-affinity constraints
- Node selector not matching
Pods Not Scaling Down#
bash
# Check HPA stabilization window
kubectl get hpa bff-hpa -n production -o yaml | grep -A10 scaleDown
# Check if PDB is blocking
kubectl get pdb -n production
# Force scale down (use carefully)
kubectl scale deployment bff -n production --replicas=3
Common causes:
- Stabilization window not elapsed
- PDB preventing scale down
- HPA metrics not available
- Minimum replicas already reached
Performance Issues After Scaling#
bash
# Check pod distribution across nodes
kubectl get pods -n production -o wide | awk '{print $7}' | sort | uniq -c
# Check for hot spots
kubectl top pods -n production --sort-by=cpu
# Verify service mesh load balancing
istioctl analyze -n production
Common causes:
- Uneven pod distribution
- Database connection limits
- Downstream service bottleneck
- Cache not warming up
Database Connection Exhaustion#
bash
# Check current connections
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT count(*) FROM pg_stat_activity"
# Per-service connection usage
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name"
# Increase connection pool
kubectl edit configmap pgbouncer-config -n production
# Update max_client_conn and default_pool_size
kubectl rollout restart deployment pgbouncer -n production
Quick Reference#
Scaling Commands#
| Action | Command |
|---|---|
| Scale deployment | kubectl scale deploy/<name> -n production --replicas=N |
| Check HPA | kubectl get hpa -n production |
| View pods | kubectl get pods -n production -o wide |
| Check capacity | kubectl top nodes |
| View pending | kubectl get pods --field-selector=status.phase=Pending |
Emergency Contacts#
| Role | Contact |
|---|---|
| Platform On-Call | #platform-oncall Slack |
| DBA On-Call | #dba-oncall Slack |
| SRE Lead | @sre-lead |