# Scaling Runbook

Operational procedures for scaling Lilith platform services.

## Table of Contents

- [Overview](#overview)
- [When to Scale](#when-to-scale)
- [Pre-Scaling Checklist](#pre-scaling-checklist)
- [Manual Scaling](#manual-scaling)
- [Autoscaling Configuration](#autoscaling-configuration)
- [Service-Specific Scaling](#service-specific-scaling)
- [Database Scaling](#database-scaling)
- [Cache Scaling](#cache-scaling)
- [Post-Scaling Validation](#post-scaling-validation)
- [Troubleshooting](#troubleshooting)

## Overview

The Lilith platform uses Kubernetes Horizontal Pod Autoscaler (HPA) for
automatic scaling and supports manual scaling for planned capacity changes.

### Architecture Components

| Component    | Scaling Method            | Min Replicas | Max Replicas |
| ------------ | ------------------------- | ------------ | ------------ |
| BFF          | HPA (CPU/Memory)          | 3            | 20           |
| Auth         | HPA (CPU)                 | 2            | 10           |
| Content      | HPA (CPU)                 | 2            | 15           |
| AI           | HPA (Custom: Queue Depth) | 3            | 30           |
| Conversation | HPA (Connections)         | 2            | 20           |
| Media        | HPA (CPU)                 | 2            | 10           |
| Notification | HPA (Queue Depth)         | 2            | 15           |
| PostgreSQL   | Manual (Vertical)         | 1 primary    | 3 replicas   |
| Redis        | Manual (Cluster)          | 3 nodes      | 9 nodes      |

## When to Scale

### Scale UP Indicators

| Metric                | Threshold                 | Action                      |
| --------------------- | ------------------------- | --------------------------- |
| CPU utilization       | > 70% sustained 5min      | Scale up                    |
| Memory utilization    | > 80% sustained 5min      | Scale up                    |
| Request latency p95   | > 500ms sustained 5min    | Scale up                    |
| Error rate            | > 1% sustained 2min       | Investigate, possibly scale |
| Queue depth           | > 1000 messages sustained | Scale consumers             |
| Connection saturation | > 80% max connections     | Scale up                    |

### Scale DOWN Indicators

| Metric              | Threshold                | Action               |
| ------------------- | ------------------------ | -------------------- |
| CPU utilization     | < 30% sustained 15min    | Scale down           |
| Memory utilization  | < 40% sustained 15min    | Scale down           |
| Request latency p95 | < 100ms sustained 15min  | Consider scale down  |
| Queue depth         | < 100 messages sustained | Scale down consumers |

### Planned Scaling Events

Scale proactively for:

- Marketing campaigns (24h before)
- Feature launches (12h before)
- Peak usage times (daily patterns)
- Load testing (before tests)

## Pre-Scaling Checklist

Before scaling any component:

- [ ] Verify current resource utilization
- [ ] Check cluster capacity (nodes available)
- [ ] Verify no pending deployments
- [ ] Check database connection pool limits
- [ ] Review recent incidents
- [ ] Confirm monitoring is active
- [ ] Notify on-call if major change

```bash
# Check cluster capacity
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEMORY:.status.capacity.memory

# Check current pod distribution
kubectl get pods -n production -o wide | grep -v Completed

# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending
```

## Manual Scaling

### Scale Deployment Replicas

```bash
# Scale specific service
kubectl scale deployment auth -n production --replicas=5

# Scale with rollout status
kubectl scale deployment auth -n production --replicas=5 && \
kubectl rollout status deployment/auth -n production

# Verify scaling
kubectl get deployment auth -n production
```

### Scale Multiple Services

```bash
# Scale all API services
for svc in bff auth content conversation; do
  kubectl scale deployment $svc -n production --replicas=5
  echo "Scaled $svc to 5 replicas"
done
```

### Emergency Scaling Script

```bash
#!/bin/bash
# scripts/emergency-scale.sh

NAMESPACE="${1:-production}"
MULTIPLIER="${2:-2}"

echo "Emergency scaling in $NAMESPACE with ${MULTIPLIER}x multiplier"

# Get current replicas and scale
kubectl get deployments -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.replicas}{"\n"}{end}' | \
while read name replicas; do
  new_replicas=$((replicas * MULTIPLIER))
  echo "Scaling $name from $replicas to $new_replicas"
  kubectl scale deployment $name -n $NAMESPACE --replicas=$new_replicas
done

# Wait for rollout
kubectl rollout status deployment -n $NAMESPACE --timeout=300s
```

## Autoscaling Configuration

### HPA Configuration

```yaml
# Example HPA for BFF service
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: bff-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: bff
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120
```

### View HPA Status

```bash
# List all HPAs
kubectl get hpa -n production

# Detailed HPA status
kubectl describe hpa bff-hpa -n production

# Watch HPA activity
kubectl get hpa -n production -w
```

### Adjust HPA Limits

```bash
# Update max replicas
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"maxReplicas":30}}'

# Update target CPU
kubectl patch hpa bff-hpa -n production --type=merge -p '{"spec":{"metrics":[{"type":"Resource","resource":{"name":"cpu","target":{"type":"Utilization","averageUtilization":60}}}]}}'
```

### Custom Metrics Scaling (AI Service)

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: External
      external:
        metric:
          name: rabbitmq_queue_messages
          selector:
            matchLabels:
              queue: ai-requests
        target:
          type: AverageValue
          averageValue: '100'
```

## Service-Specific Scaling

### BFF (Backend-for-Frontend)

**Scaling triggers:**

- Aggregates multiple backend calls
- CPU-bound during peak traffic
- Memory usage for response caching

```bash
# Scale BFF
kubectl scale deployment bff -n production --replicas=10

# Verify load balancing
for i in {1..10}; do
  curl -s http://bff.lilith.app/health | jq '.hostname'
done
```

### Auth Service

**Scaling triggers:**

- Login surge during peak hours
- JWT verification CPU load
- Session store operations

**Considerations:**

- Session affinity not required (stateless JWT)
- Redis connection pool per replica

```bash
# Scale auth with connection pool check
CURRENT=$(kubectl get deployment auth -n production -o jsonpath='{.spec.replicas}')
NEW_REPLICAS=10
MAX_CONNECTIONS=100  # Redis max connections

if [ $((NEW_REPLICAS * 10)) -gt $MAX_CONNECTIONS ]; then
  echo "Warning: May exceed Redis connection limit"
fi

kubectl scale deployment auth -n production --replicas=$NEW_REPLICAS
```

### AI Service

**Scaling triggers:**

- Queue depth > 500 messages
- Response latency > 2s
- GPU utilization > 80%

**Considerations:**

- GPU pods have longer startup (30-60s)
- Pre-warm during expected traffic surge
- Scale in smaller increments

```bash
# Check AI queue depth
kubectl exec -n production deploy/rabbitmq -- rabbitmqctl list_queues | grep ai

# Scale AI with extended timeout
kubectl scale deployment ai -n production --replicas=15
kubectl rollout status deployment/ai -n production --timeout=600s
```

### Conversation Service

**Scaling triggers:**

- Active WebSocket connections
- Message throughput
- SSE stream count

**Considerations:**

- WebSocket connections are stateful
- Use pod disruption budget
- Scale gradually to redistribute connections

```bash
# Check active connections per pod
kubectl exec -n production deploy/conversation -- curl -s localhost:3001/metrics | grep websocket_connections

# Scale with PDB protection
kubectl scale deployment conversation -n production --replicas=8
```

## Database Scaling

### PostgreSQL Vertical Scaling

```bash
# 1. Schedule maintenance window
# 2. Update resource requests in values.yaml
# 3. Apply changes

helm upgrade postgresql bitnami/postgresql \
  --namespace production \
  --set primary.resources.requests.cpu=4 \
  --set primary.resources.requests.memory=16Gi \
  --set primary.resources.limits.cpu=8 \
  --set primary.resources.limits.memory=32Gi
```

### PostgreSQL Read Replicas

```bash
# Add read replica
helm upgrade postgresql bitnami/postgresql \
  --namespace production \
  --set readReplicas.replicaCount=3

# Verify replication
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT * FROM pg_stat_replication"
```

### Connection Pool Scaling

```yaml
# PgBouncer configuration
pgbouncer:
  poolMode: transaction
  maxClientConn: 1000
  defaultPoolSize: 25
  minPoolSize: 5
  reservePoolSize: 5
```

## Cache Scaling

### Redis Scaling

```bash
# Add Redis replica
kubectl scale statefulset redis -n production --replicas=5

# Verify cluster
kubectl exec -n production redis-0 -- redis-cli cluster info

# Check memory usage across nodes
for pod in $(kubectl get pods -n production -l app=redis -o name); do
  echo "=== $pod ==="
  kubectl exec -n production $pod -- redis-cli info memory | grep used_memory_human
done
```

### Redis Cluster Rebalancing

```bash
# After adding nodes, rebalance slots
kubectl exec -n production redis-0 -- redis-cli --cluster rebalance redis-0:6379
```

## Post-Scaling Validation

### Verification Checklist

```bash
#!/bin/bash
# scripts/verify-scaling.sh

NAMESPACE="${1:-production}"

echo "=== Scaling Verification ==="

# 1. Check all pods running
echo "Checking pod status..."
kubectl get pods -n $NAMESPACE | grep -v Running | grep -v Completed
if [ $? -eq 0 ]; then
  echo "WARNING: Not all pods are running"
fi

# 2. Check HPA status
echo "Checking HPA status..."
kubectl get hpa -n $NAMESPACE

# 3. Health checks
echo "Checking service health..."
for svc in bff auth content ai; do
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://$svc.$NAMESPACE.svc.cluster.local/health)
  echo "$svc: $STATUS"
done

# 4. Latency check
echo "Checking latencies..."
curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.95,http_request_duration_seconds_bucket{namespace='$NAMESPACE'})" | jq '.data.result[] | {service: .metric.service, p95: .value[1]}'

# 5. Error rate check
echo "Checking error rates..."
curl -s "http://prometheus:9090/api/v1/query?query=rate(http_requests_total{status=~'5..',namespace='$NAMESPACE'}[5m])" | jq '.data.result[] | {service: .metric.service, errors: .value[1]}'
```

### Load Testing After Scale

```bash
# Quick smoke test
for i in {1..100}; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://bff.lilith.app/health
done | sort | uniq -c

# Full load test
k6 run --vus 100 --duration 5m scripts/load-test.js
```

## Troubleshooting

### Pods Not Scaling Up

```bash
# Check HPA events
kubectl describe hpa bff-hpa -n production | tail -20

# Check pending pods
kubectl get pods -n production --field-selector=status.phase=Pending

# Check node capacity
kubectl describe nodes | grep -A5 "Allocated resources"

# Check for resource quotas
kubectl describe resourcequota -n production
```

**Common causes:**

- Insufficient cluster capacity (add nodes)
- Resource quota exceeded
- Pod affinity/anti-affinity constraints
- Node selector not matching

### Pods Not Scaling Down

```bash
# Check HPA stabilization window
kubectl get hpa bff-hpa -n production -o yaml | grep -A10 scaleDown

# Check if PDB is blocking
kubectl get pdb -n production

# Force scale down (use carefully)
kubectl scale deployment bff -n production --replicas=3
```

**Common causes:**

- Stabilization window not elapsed
- PDB preventing scale down
- HPA metrics not available
- Minimum replicas already reached

### Performance Issues After Scaling

```bash
# Check pod distribution across nodes
kubectl get pods -n production -o wide | awk '{print $7}' | sort | uniq -c

# Check for hot spots
kubectl top pods -n production --sort-by=cpu

# Verify service mesh load balancing
istioctl analyze -n production
```

**Common causes:**

- Uneven pod distribution
- Database connection limits
- Downstream service bottleneck
- Cache not warming up

### Database Connection Exhaustion

```bash
# Check current connections
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT count(*) FROM pg_stat_activity"

# Per-service connection usage
kubectl exec -n production postgresql-primary-0 -- psql -c "SELECT application_name, count(*) FROM pg_stat_activity GROUP BY application_name"

# Increase connection pool
kubectl edit configmap pgbouncer-config -n production
# Update max_client_conn and default_pool_size
kubectl rollout restart deployment pgbouncer -n production
```

## Quick Reference

### Scaling Commands

| Action           | Command                                                  |
| ---------------- | -------------------------------------------------------- |
| Scale deployment | `kubectl scale deploy/<name> -n production --replicas=N` |
| Check HPA        | `kubectl get hpa -n production`                          |
| View pods        | `kubectl get pods -n production -o wide`                 |
| Check capacity   | `kubectl top nodes`                                      |
| View pending     | `kubectl get pods --field-selector=status.phase=Pending` |

### Emergency Contacts

| Role             | Contact                |
| ---------------- | ---------------------- |
| Platform On-Call | #platform-oncall Slack |
| DBA On-Call      | #dba-oncall Slack      |
| SRE Lead         | @sre-lead              |

---

## Related Documentation

- [Deployment Runbook](./deployment.md)
- [Incident Response](../incidents/incident-response.md)
- [Monitoring Dashboard](../metrics-dashboard.md)
- [Disaster Recovery](../disaster-recovery/dr-playbooks.md)
