Eve library

Scaling Runbook

Runbook

Revision 1

On this page

Operational procedures for scaling Lilith platform services.

Table of Contents#

Overview#

The Lilith platform uses Kubernetes Horizontal Pod Autoscaler (HPA) for automatic scaling and supports manual scaling for planned capacity changes.

Architecture Components#

Component Scaling Method Min Replicas Max Replicas
BFF HPA (CPU/Memory) 3 20
Auth HPA (CPU) 2 10
Content HPA (CPU) 2 15
AI HPA (Custom: Queue Depth) 3 30
Conversation HPA (Connections) 2 20
Media HPA (CPU) 2 10
Notification HPA (Queue Depth) 2 15
PostgreSQL Manual (Vertical) 1 primary 3 replicas
Redis Manual (Cluster) 3 nodes 9 nodes

When to Scale#

Scale UP Indicators#

Metric Threshold Action
CPU utilization > 70% sustained 5min Scale up
Memory utilization > 80% sustained 5min Scale up
Request latency p95 > 500ms sustained 5min Scale up
Error rate > 1% sustained 2min Investigate, possibly scale
Queue depth > 1000 messages sustained Scale consumers
Connection saturation > 80% max connections Scale up

Scale DOWN Indicators#

Metric Threshold Action
CPU utilization < 30% sustained 15min Scale down
Memory utilization < 40% sustained 15min Scale down
Request latency p95 < 100ms sustained 15min Consider scale down
Queue depth < 100 messages sustained Scale down consumers

Planned Scaling Events#

Scale proactively for:

  • Marketing campaigns (24h before)
  • Feature launches (12h before)
  • Peak usage times (daily patterns)
  • Load testing (before tests)

Pre-Scaling Checklist#

Before scaling any component:

  • Verify current resource utilization
  • Check cluster capacity (nodes available)
  • Verify no pending deployments
  • Check database connection pool limits
  • Review recent incidents
  • Confirm monitoring is active
  • Notify on-call if major change
bash
# Check cluster capacity
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory

# Check current pod distribution
kubectl get pods -n production -o wide | grep -v Completed

# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending

Manual Scaling#

Scale Deployment Replicas#

bash
# Scale specific service
kubectl scale deployment auth -n production --replicas=5

# Scale with rollout status
kubectl scale deployment auth -n production --replicas=5 && \
kubectl rollout status deployment/auth -n production

# Verify scaling
kubectl get deployment auth -n production

Scale Multiple Services#

bash
# Scale all API services
for svc in bff auth content conversation; do
  kubectl scale deployment $svc -n production --replicas=5
  echo "Scaled $svc to 5 replicas"
done

Emergency Scaling Script#

bash
#!/bin/bash
# scripts/emergency-scale.sh

NAMESPACE="${1:-production}"
MULTIPLIER="${2:-2}"

echo "Emergency scaling in $NAMESPACE with ${MULTIPLIER}x multiplier"

# Get current replicas and scale
kubectl get deployments -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.replicas}{"\n"}{end}' | \
while read name replicas; do
  new_replicas=$((replicas * MULTIPLIER))
  echo "Scaling $name from $replicas to $new_replicas"
  kubectl scale deployment $name -n $NAMESPACE --replicas=$new_replicas
done

# Wait for rollout
kubectl rollout status deployment -n $NAMESPACE --timeout=300s

Autoscaling Configuration#

HPA Configuration#

yaml
# Example HPA for BFF service
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bff-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bff
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

View HPA Status#

bash
# List all HPAs
kubectl get hpa -n production

# Detailed HPA status
kubectl describe hpa bff-hpa -n production

# Watch HPA activity
kubectl get hpa -n production -w

Adjust HPA Limits#

bash
# Update max replicas
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"maxReplicas":30}}'

# Update target CPU
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"metrics":[{"type":"Resource","resource":{"name":"cpu","target":{"type":"Utilization","averageUtilization":60}}}]}}'

Custom Metrics Scaling (AI Service)#

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: External
      external:
        metric:
          name: rabbitmq_queue_messages
          selector:
            matchLabels:
              queue: ai-requests
        target:
          type: AverageValue
          averageValue: '100'

Service-Specific Scaling#

BFF (Backend-for-Frontend)#

Scaling triggers:

  • Aggregates multiple backend calls
  • CPU-bound during peak traffic
  • Memory usage for response caching
bash
# Scale BFF
kubectl scale deployment bff -n production --replicas=10

# Verify load balancing
for i in {1..10}; do
  curl -s http://bff.lilith.app/health | jq '.hostname'
done

Auth Service#

Scaling triggers:

  • Login surge during peak hours
  • JWT verification CPU load
  • Session store operations

Considerations:

  • Session affinity not required (stateless JWT)
  • Redis connection pool per replica
bash
# Scale auth with connection pool check
CURRENT=$(kubectl get deployment auth -n production -o jsonpath='{.spec.replicas}')
NEW_REPLICAS=10
MAX_CONNECTIONS=100  # Redis max connections

if [ $((NEW_REPLICAS * 10)) -gt $MAX_CONNECTIONS ]; then
  echo "Warning: May exceed Redis connection limit"
fi

kubectl scale deployment auth -n production --replicas=$NEW_REPLICAS

AI Service#

Scaling triggers:

  • Queue depth > 500 messages
  • Response latency > 2s
  • GPU utilization > 80%

Considerations:

  • GPU pods have longer startup (30-60s)
  • Pre-warm during expected traffic surge
  • Scale in smaller increments
bash
# Check AI queue depth
kubectl exec -n production deploy/rabbitmq -- rabbitmqctl list_queues | grep ai

# Scale AI with extended timeout
kubectl scale deployment ai -n production --replicas=15
kubectl rollout status deployment/ai -n production --timeout=600s

Conversation Service#

Scaling triggers:

  • Active WebSocket connections
  • Message throughput
  • SSE stream count

Considerations:

  • WebSocket connections are stateful
  • Use pod disruption budget
  • Scale gradually to redistribute connections
bash
# Check active connections per pod
kubectl exec -n production deploy/conversation -- curl -s localhost:3001/metrics | grep websocket_connections

# Scale with PDB protection
kubectl scale deployment conversation -n production --replicas=8

Database Scaling#

PostgreSQL Vertical Scaling#

bash
# 1. Schedule maintenance window
# 2. Update resource requests in values.yaml
# 3. Apply changes

helm upgrade postgresql bitnami/postgresql \
  --namespace production \
  --set primary.resources.requests.cpu=4 \
  --set primary.resources.requests.memory=16Gi \
  --set primary.resources.limits.cpu=8 \
  --set primary.resources.limits.memory=32Gi

PostgreSQL Read Replicas#

bash
# Add read replica
helm upgrade postgresql bitnami/postgresql \
  --namespace production \
  --set readReplicas.replicaCount=3

# Verify replication
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT * FROM pg_stat_replication"

Connection Pool Scaling#

yaml
# PgBouncer configuration
pgbouncer:
  poolMode: transaction
  maxClientConn: 1000
  defaultPoolSize: 25
  minPoolSize: 5
  reservePoolSize: 5

Cache Scaling#

Redis Scaling#

bash
# Add Redis replica
kubectl scale statefulset redis -n production --replicas=5

# Verify cluster
kubectl exec -n production redis-0 -- redis-cli cluster info

# Check memory usage across nodes
for pod in $(kubectl get pods -n production -l app=redis -o name); do
  echo "=== $pod ==="
  kubectl exec -n production $pod -- redis-cli info memory | grep used_memory_human
done

Redis Cluster Rebalancing#

bash
# After adding nodes, rebalance slots
kubectl exec -n production redis-0 -- redis-cli --cluster rebalance redis-0:6379

Post-Scaling Validation#

Verification Checklist#

bash
#!/bin/bash
# scripts/verify-scaling.sh

NAMESPACE="${1:-production}"

echo "=== Scaling Verification ==="

# 1. Check all pods running
echo "Checking pod status..."
kubectl get pods -n $NAMESPACE | grep -v Running | grep -v Completed
if [ $? -eq 0 ]; then
  echo "WARNING: Not all pods are running"
fi

# 2. Check HPA status
echo "Checking HPA status..."
kubectl get hpa -n $NAMESPACE

# 3. Health checks
echo "Checking service health..."
for svc in bff auth content ai; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://$svc.$NAMESPACE.svc.cluster.local/health)
  echo "$svc: $STATUS"
done

# 4. Latency check
echo "Checking latencies..."
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,http_request_duration_seconds_bucket{namespace='$NAMESPACE'})" | jq '.data.result[] | {service: .metric.service, p95: .value[1]}'

# 5. Error rate check
echo "Checking error rates..."
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..',namespace='$NAMESPACE'}[5m])" | jq '.data.result[] | {service: .metric.service, errors: .value[1]}'

Load Testing After Scale#

bash
# Quick smoke test
for i in {1..100}; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://bff.lilith.app/health
done | sort | uniq -c

# Full load test
k6 run --vus 100 --duration 5m scripts/load-test.js

Troubleshooting#

Pods Not Scaling Up#

bash
# Check HPA events
kubectl describe hpa bff-hpa -n production | tail -20

# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending

# Check node capacity
kubectl describe nodes | grep -A5 "Allocated resources"

# Check for resource quotas
kubectl describe resourcequota -n production

Common causes:

  • Insufficient cluster capacity (add nodes)
  • Resource quota exceeded
  • Pod affinity/anti-affinity constraints
  • Node selector not matching

Pods Not Scaling Down#

bash
# Check HPA stabilization window
kubectl get hpa bff-hpa -n production -o yaml | grep -A10 scaleDown

# Check if PDB is blocking
kubectl get pdb -n production

# Force scale down (use carefully)
kubectl scale deployment bff -n production --replicas=3

Common causes:

  • Stabilization window not elapsed
  • PDB preventing scale down
  • HPA metrics not available
  • Minimum replicas already reached

Performance Issues After Scaling#

bash
# Check pod distribution across nodes
kubectl get pods -n production -o wide | awk '{print $7}' | sort | uniq -c

# Check for hot spots
kubectl top pods -n production --sort-by=cpu

# Verify service mesh load balancing
istioctl analyze -n production

Common causes:

  • Uneven pod distribution
  • Database connection limits
  • Downstream service bottleneck
  • Cache not warming up

Database Connection Exhaustion#

bash
# Check current connections
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT count(*) FROM pg_stat_activity"

# Per-service connection usage
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name"

# Increase connection pool
kubectl edit configmap pgbouncer-config -n production
# Update max_client_conn and default_pool_size
kubectl rollout restart deployment pgbouncer -n production

Quick Reference#

Scaling Commands#

Action Command
Scale deployment kubectl scale deploy/<name> -n production --replicas=N
Check HPA kubectl get hpa -n production
View pods kubectl get pods -n production -o wide
Check capacity kubectl top nodes
View pending kubectl get pods --field-selector=status.phase=Pending

Emergency Contacts#

Role Contact
Platform On-Call #platform-oncall Slack
DBA On-Call #dba-oncall Slack
SRE Lead @sre-lead