Eve library

Capacity Planning Guide

Runbook

Revision 1

On this page

Purpose: Document expected load profiles, resource requirements, and scaling strategies for Lilith Platform services to ensure adequate capacity during launch and growth phases.

Table of Contents#

  1. Load Profiles
  2. Service Resource Requirements
  3. Scaling Thresholds
  4. Database Capacity
  5. AI Service Capacity
  6. Cost Projections
  7. Capacity Planning Process

Load Profiles#

User Growth Projections#

Phase Timeline Daily Active Users (DAU) Peak Concurrent Requests/Second
Soft Launch Week 1-2 1,000 - 5,000 500 50
Limited Release Week 3-4 5,000 - 25,000 2,500 250
Public Launch Week 5-8 25,000 - 100,000 10,000 1,000
Growth Phase Month 3-6 100,000 - 500,000 50,000 5,000
Scale Phase Month 6-12 500,000 - 2,000,000 200,000 20,000

Traffic Patterns#

Daily Pattern (UTC)#

text
Hour    | Load Factor | Description
--------|-------------|------------------------------------------
00-06   | 0.3x       | Off-peak (Americas night)
06-09   | 0.7x       | Morning meditation peak (EU)
09-12   | 0.8x       | Work break sessions
12-15   | 0.6x       | Midday lull
15-18   | 1.0x       | Peak (EU evening + Americas afternoon)
18-22   | 1.2x       | Maximum peak (Americas evening)
22-00   | 0.8x       | Evening wind-down

Weekly Pattern#

  • Monday: 110% baseline (New week motivation)
  • Tuesday-Thursday: 100% baseline
  • Friday: 90% baseline
  • Saturday: 85% baseline
  • Sunday: 120% baseline (Self-care day)

Event-Driven Spikes#

Event Type Expected Multiplier Duration Planning Buffer
Marketing Campaign 3-5x 24-48 hours Pre-scale 2x
PR/Media Feature 5-10x 12-24 hours Auto-scale aggressive
App Store Feature 10-20x 48-72 hours Pre-scale 5x
Viral Moment 10-50x Variable Emergency scaling

Service Resource Requirements#

Core Services#

BFF (Backend for Frontend)#

yaml
Base Resources:
  cpu: 500m
  memory: 512Mi
  replicas: 3

Per 1000 RPS:
  cpu: 1000m
  memory: 768Mi
  replicas: +2

Scaling Limits:
  min_replicas: 3
  max_replicas: 50
  target_cpu: 70%
  target_memory: 80%

Auth Service#

yaml
Base Resources:
  cpu: 250m
  memory: 256Mi
  replicas: 3

Per 1000 RPS:
  cpu: 500m
  memory: 384Mi
  replicas: +1

Scaling Limits:
  min_replicas: 3
  max_replicas: 20
  target_cpu: 60%
  target_memory: 70%

Content Service#

yaml
Base Resources:
  cpu: 500m
  memory: 512Mi
  replicas: 3

Per 1000 RPS:
  cpu: 750m
  memory: 640Mi
  replicas: +2

Scaling Limits:
  min_replicas: 3
  max_replicas: 30
  target_cpu: 70%
  target_memory: 75%

Conversation Service#

yaml
Base Resources:
  cpu: 500m
  memory: 512Mi
  replicas: 3

Per 1000 Active Conversations:
  cpu: 250m
  memory: 128Mi
  replicas: +1

Scaling Limits:
  min_replicas: 3
  max_replicas: 25
  target_cpu: 65%
  target_memory: 75%

Media Service#

yaml
Base Resources:
  cpu: 1000m
  memory: 1Gi
  replicas: 3

Per 1000 Media Requests:
  cpu: 500m
  memory: 512Mi
  replicas: +1

Scaling Limits:
  min_replicas: 3
  max_replicas: 40
  target_cpu: 75%
  target_memory: 80%

AI Service#

yaml
Base Resources:
  cpu: 2000m
  memory: 4Gi
  replicas: 5

Per 100 AI Requests/min:
  cpu: 1000m
  memory: 2Gi
  replicas: +2

Scaling Limits:
  min_replicas: 5
  max_replicas: 50
  target_cpu: 60%
  target_memory: 70%

GPU Resources (if applicable):
  gpu: 1x NVIDIA T4
  gpu_memory: 16Gi

Notification Service#

yaml
Base Resources:
  cpu: 250m
  memory: 256Mi
  replicas: 2

Per 10000 Notifications/hour:
  cpu: 250m
  memory: 128Mi
  replicas: +1

Scaling Limits:
  min_replicas: 2
  max_replicas: 15
  target_cpu: 70%
  target_memory: 80%

Moderation Service#

yaml
Base Resources:
  cpu: 500m
  memory: 512Mi
  replicas: 2

Per 500 Moderation Requests/min:
  cpu: 250m
  memory: 256Mi
  replicas: +1

Scaling Limits:
  min_replicas: 2
  max_replicas: 20
  target_cpu: 65%
  target_memory: 75%

Scaling Thresholds#

Horizontal Pod Autoscaler (HPA) Configuration#

yaml
# Standard service HPA template
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ${SERVICE_NAME}-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ${SERVICE_NAME}
  minReplicas: ${MIN_REPLICAS}
  maxReplicas: ${MAX_REPLICAS}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: ${CPU_TARGET}
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: ${MEMORY_TARGET}
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: ${RPS_TARGET}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
      selectPolicy: Max

Scaling Trigger Matrix#

Metric Warning Scale Up Scale Down Critical
CPU Utilization 60% 70% <40% for 5m 90%
Memory Utilization 70% 80% <50% for 5m 90%
Request Latency P95 200ms 300ms <100ms 500ms
Request Queue Depth 50 100 <20 500
Error Rate 0.1% 0.5% <0.01% 1%
Active Connections 80% max 90% max <50% max 95% max

Pre-scaling Commands#

bash
# Pre-scale for anticipated traffic surge
kubectl scale deployment bff --replicas=10 -n production
kubectl scale deployment content --replicas=8 -n production
kubectl scale deployment ai --replicas=15 -n production

# Verify scaling
kubectl get hpa -n production

# Scale node pool (GKE example)
gcloud container clusters resize lilith-production \
  --node-pool=general-pool \
  --num-nodes=20 \
  --zone=us-central1-a

Database Capacity#

PostgreSQL#

Connection Pool Sizing#

text
Connections per service replica: 10
Service replicas (max): 200
Safety buffer: 20%

Recommended max_connections: 200 * 10 * 1.2 = 2,400

PgBouncer pool sizing:
- default_pool_size: 20
- max_client_conn: 5000
- reserve_pool_size: 5

Storage Projections#

Data Type Per User 100K Users 1M Users Growth/Month
User profiles 50KB 5GB 50GB 5%
Sessions 10KB 1GB 10GB 10%
Content metadata 100KB 10GB 100GB 3%
Analytics 500KB 50GB 500GB 15%
Audit logs 200KB 20GB 200GB 8%
Total 860KB 86GB 860GB ~10%

Instance Sizing#

Phase Instance Type vCPU RAM Storage IOPS
Soft Launch db.r6g.large 2 16GB 100GB 3000
Public Launch db.r6g.xlarge 4 32GB 500GB 6000
Growth db.r6g.2xlarge 8 64GB 1TB 12000
Scale db.r6g.4xlarge 16 128GB 2TB 24000

Redis#

Memory Sizing#

text
Session data: 5KB/session * 200,000 concurrent = 1GB
Cache data: 50KB/user * 100,000 DAU = 5GB
Rate limiting: 1KB/IP * 500,000 IPs = 500MB
Pub/Sub channels: 100MB

Total: ~7GB (allocate 16GB for buffer)

Cluster Configuration#

Phase Nodes Node Type Total Memory Shards
Launch 3 cache.r6g.large 39GB 1
Growth 6 cache.r6g.xlarge 156GB 2
Scale 9 cache.r6g.2xlarge 468GB 3

Vector Database (Milvus)#

Collection Sizing#

text
Embedding dimensions: 1536 (OpenAI)
Vectors per user: ~100
Index type: IVF_FLAT

Per 100K users:
- Raw vectors: 100K * 100 * 1536 * 4 bytes = 58GB
- Index overhead: 20%
- Total: ~70GB

AI Service Capacity#

LLM Token Budget#

Daily Token Projections#

User Tier Tokens/Session Sessions/Day Daily Tokens
Free 2,000 1 2,000
Premium 5,000 3 15,000
Pro 10,000 5 50,000

Monthly Token Budget (100K DAU)#

text
Free users (70%): 70,000 * 2,000 * 30 = 4.2B tokens
Premium (25%): 25,000 * 15,000 * 30 = 11.25B tokens
Pro (5%): 5,000 * 50,000 * 30 = 7.5B tokens

Total: ~23B tokens/month

Rate Limiting Configuration#

yaml
ai_rate_limits:
  free_tier:
    requests_per_minute: 5
    tokens_per_day: 5000
    concurrent_requests: 1
  premium_tier:
    requests_per_minute: 20
    tokens_per_day: 50000
    concurrent_requests: 3
  pro_tier:
    requests_per_minute: 60
    tokens_per_day: 200000
    concurrent_requests: 5

AI Queue Management#

yaml
queue_config:
  high_priority: # Premium/Pro users
    max_wait_time: 5s
    workers: 30
  normal_priority: # Free users
    max_wait_time: 30s
    workers: 20
  batch_priority: # Background processing
    max_wait_time: 300s
    workers: 10

Cost Projections#

Infrastructure Costs (AWS)#

Phase Compute Database Cache Storage Network Total/Month
Soft Launch $2,000 $500 $200 $100 $200 $3,000
Public Launch $8,000 $2,000 $800 $500 $1,000 $12,300
Growth (100K) $20,000 $5,000 $2,000 $2,000 $3,000 $32,000
Scale (500K) $60,000 $15,000 $6,000 $8,000 $10,000 $99,000
Scale (1M) $100,000 $25,000 $10,000 $15,000 $20,000 $170,000

AI API Costs#

Provider Model Price/1M Tokens 100K DAU Monthly 1M DAU Monthly
OpenAI GPT-4 $30 input/$60 output $690,000 $6,900,000
OpenAI GPT-4o-mini $0.15 input/$0.60 output $8,625 $86,250
Anthropic Claude 3.5 $3 input/$15 output $138,000 $1,380,000
Anthropic Claude 3 Haiku $0.25 input/$1.25 output $11,500 $115,000

Recommendation: Use tiered model selection:

  • Free tier: Claude 3 Haiku / GPT-4o-mini
  • Premium: Claude 3.5 Sonnet / GPT-4o
  • Pro: Claude 3 Opus / GPT-4

Cost Per User Metrics#

Phase DAU Total Monthly Cost Cost/DAU Cost/MAU
Soft Launch 5K $15,000 $3.00 $0.50
Public Launch 50K $50,000 $1.00 $0.17
Growth 100K $100,000 $1.00 $0.17
Scale 500K $400,000 $0.80 $0.13
Scale 1M $700,000 $0.70 $0.12

Capacity Planning Process#

Weekly Capacity Review#

  1. Review Metrics

    bash
    # Generate capacity report
    ./scripts/generate-capacity-report.sh --period=7d
    
  2. Check Growth Trends

    • DAU growth rate
    • Request rate trends
    • Resource utilization trends
  3. Update Projections

    • Adjust forecasts based on actual data
    • Identify potential bottlenecks
  4. Plan Adjustments

    • Schedule scaling changes
    • Budget adjustments if needed

Pre-Launch Capacity Checklist#

  • Load test at 2x expected launch traffic
  • Verify auto-scaling triggers work correctly
  • Pre-scale services to handle initial surge
  • Confirm database connections sufficient
  • Validate CDN capacity for media
  • Test failover procedures under load
  • Verify AI rate limits prevent budget overrun
  • Confirm monitoring alerts configured
  • Document emergency scaling procedures
  • Brief on-call team on capacity thresholds

Emergency Scaling Procedures#

bash
# Emergency scale-up script
#!/bin/bash
NAMESPACE=${1:-production}
MULTIPLIER=${2:-2}

# Scale all deployments
for deploy in $(kubectl get deploy -n $NAMESPACE -o name); do
  current=$(kubectl get $deploy -n $NAMESPACE -o jsonpath='{.spec.replicas}')
  new=$((current * MULTIPLIER))
  kubectl scale $deploy --replicas=$new -n $NAMESPACE
done

# Scale node pool
gcloud container clusters resize lilith-production \
  --node-pool=general-pool \
  --num-nodes=$(($(gcloud container clusters describe lilith-production --format='value(nodePools[0].initialNodeCount)') * MULTIPLIER)) \
  --zone=us-central1-a \
  --quiet

echo "Emergency scale completed. Verify with: kubectl get pods -n $NAMESPACE"

Capacity Monitoring Queries#

promql
# Service capacity utilization
sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)
  / sum(kube_pod_container_resource_limits{resource="cpu",namespace="production"}) by (pod) * 100

# Database connection utilization
pg_stat_activity_count / pg_settings_max_connections * 100

# Redis memory utilization
redis_memory_used_bytes / redis_memory_max_bytes * 100

# Request headroom
1 - (sum(rate(http_requests_total[5m])) / sum(service_max_rps))

Appendix: Quick Reference#

Scaling Cheat Sheet#

Scenario Action Command
CPU > 80% Scale pods kubectl scale deploy X --replicas=+2
Memory > 85% Scale pods + add nodes Scale + gcloud clusters resize
Latency > 300ms Scale + check DB Scale + check slow queries
Queue depth > 100 Scale workers Increase HPA max
DB connections > 80% Add PgBouncer capacity Scale PgBouncer

Key Thresholds Summary#

Resource Warning Action Critical
CPU 60% 70% 90%
Memory 70% 80% 90%
Disk 70% 80% 90%
Connections 70% 80% 90%
Error Rate 0.1% 0.5% 1%
P95 Latency 200ms 300ms 500ms

Last Updated: January 2026 Review Cycle: Weekly during launch, Monthly thereafter