This guide covers deploying the GPU Jobs system in production environments.
Table of Contents#
- Architecture Overview
- Prerequisites
- Deployment Options
- Infrastructure Setup
- Configuration
- Scaling Configuration
- Monitoring Setup
- Security Considerations
- Disaster Recovery
- Operations Runbook
Architecture Overview#
text
┌─────────────────────────────────────────────────────┐
│ Load Balancer │
└────────────────────────┬────────────────────────────┘
│
┌───────────────────────────────┼───────────────────────────────┐
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ API Server │ │ API Server │ │ API Server │
│ (Node.js) │ │ (Node.js) │ │ (Node.js) │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
└───────────────────────────────┼───────────────────────────────┘
│
┌──────────────────┬────────────────────────────┼────────────────────────────┬──────────────────┐
│ │ │ │ │
┌───▼───┐ ┌─────▼─────┐ ┌───────▼───────┐ ┌──────▼──────┐ ┌─────▼─────┐
│ Redis │ │ PostgreSQL│ │ Job Queue │ │ Object │ │ Monitoring│
│ Cache │ │ Database │ │ (BullMQ) │ │ Storage │ │ (Grafana) │
└───────┘ └───────────┘ └───────┬───────┘ │ (S3/MinIO) │ └───────────┘
│ └─────────────┘
┌───────────────────────────────┼───────────────────────────────┐
│ │ │
┌───────▼───────┐ ┌───────▼───────┐ ┌───────▼───────┐
│ GPU Provider │ │ GPU Provider │ │ GPU Provider │
│ (RunPod) │ │ (Modal) │ │ (AWS ECS) │
└───────────────┘ └───────────────┘ └───────────────┘
Prerequisites#
Required Services#
- PostgreSQL 15+: Job state persistence
- Redis 7+: Caching and job queue
- S3-compatible storage: MinIO, AWS S3, or GCS
- Container runtime: Docker or Kubernetes
Required Credentials#
bash
# Database
DATABASE_URL=postgresql://user:password@host:5432/yemaya_gpu_jobs
# Cache
REDIS_URL=redis://:password@host:6379
# Storage
S3_ENDPOINT=https://s3.amazonaws.com
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key
S3_BUCKET=yemaya-gpu-jobs
# GPU Providers (at least one required)
RUNPOD_API_KEY=your_runpod_key
RUNPOD_ENDPOINT_ID=your_endpoint_id
MODAL_TOKEN=your_modal_token
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
Deployment Options#
Option 1: Docker Compose (Small Scale)#
yaml
# docker-compose.prod.yml
version: '3.8'
services:
api:
image: your-registry/yemaya-api:latest
environment:
- NODE_ENV=production
- DATABASE_URL=${DATABASE_URL}
- REDIS_URL=${REDIS_URL}
- RUNPOD_API_KEY=${RUNPOD_API_KEY}
ports:
- '3000:3000'
deploy:
replicas: 3
resources:
limits:
cpus: '2'
memory: 4G
healthcheck:
test: ['CMD', 'curl', '-f', 'http://localhost:3000/health']
interval: 30s
timeout: 10s
retries: 3
workers:
image: your-registry/yemaya-workers:latest
environment:
- NODE_ENV=production
- DATABASE_URL=${DATABASE_URL}
- REDIS_URL=${REDIS_URL}
- RUNPOD_API_KEY=${RUNPOD_API_KEY}
deploy:
replicas: 2
resources:
limits:
cpus: '1'
memory: 2G
postgres:
image: postgres:15
environment:
POSTGRES_DB: yemaya_gpu_jobs
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASSWORD}
volumes:
- postgres_data:/var/lib/postgresql/data
deploy:
resources:
limits:
cpus: '2'
memory: 4G
redis:
image: redis:7-alpine
command: redis-server --requirepass ${REDIS_PASSWORD}
volumes:
- redis_data:/data
deploy:
resources:
limits:
cpus: '1'
memory: 2G
volumes:
postgres_data:
redis_data:
Deploy:
bash
docker compose -f docker-compose.prod.yml up -d
Option 2: Kubernetes (Production Scale)#
yaml
# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: gpu-jobs-api
namespace: yemaya
spec:
replicas: 3
selector:
matchLabels:
app: gpu-jobs-api
template:
metadata:
labels:
app: gpu-jobs-api
spec:
containers:
- name: api
image: your-registry/yemaya-api:latest
ports:
- containerPort: 3000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: gpu-jobs-secrets
key: database-url
- name: REDIS_URL
valueFrom:
secretKeyRef:
name: gpu-jobs-secrets
key: redis-url
- name: RUNPOD_API_KEY
valueFrom:
secretKeyRef:
name: gpu-jobs-secrets
key: runpod-api-key
resources:
requests:
cpu: '500m'
memory: '1Gi'
limits:
cpu: '2000m'
memory: '4Gi'
livenessProbe:
httpGet:
path: /health
port: 3000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health/ready
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: gpu-jobs-api
namespace: yemaya
spec:
selector:
app: gpu-jobs-api
ports:
- port: 80
targetPort: 3000
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gpu-jobs-api
namespace: yemaya
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
tls:
- hosts:
- api.yemaya.example.com
secretName: gpu-jobs-tls
rules:
- host: api.yemaya.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gpu-jobs-api
port:
number: 80
yaml
# kubernetes/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: gpu-jobs-api
namespace: yemaya
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: gpu-jobs-api
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
Deploy:
bash
kubectl apply -f kubernetes/
Option 3: AWS ECS Fargate#
json
{
"family": "gpu-jobs-api",
"networkMode": "awsvpc",
"requiresCompatibilities": ["FARGATE"],
"cpu": "1024",
"memory": "2048",
"executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",
"taskRoleArn": "arn:aws:iam::ACCOUNT:role/gpuJobsTaskRole",
"containerDefinitions": [
{
"name": "api",
"image": "ACCOUNT.dkr.ecr.REGION.amazonaws.com/yemaya-api:latest",
"portMappings": [
{
"containerPort": 3000,
"protocol": "tcp"
}
],
"environment": [{ "name": "NODE_ENV", "value": "production" }],
"secrets": [
{
"name": "DATABASE_URL",
"valueFrom": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:gpu-jobs/database-url"
},
{
"name": "RUNPOD_API_KEY",
"valueFrom": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:gpu-jobs/runpod-key"
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/gpu-jobs-api",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
},
"healthCheck": {
"command": [
"CMD-SHELL",
"curl -f http://localhost:3000/health || exit 1"
],
"interval": 30,
"timeout": 5,
"retries": 3
}
}
]
}
Infrastructure Setup#
Database Setup#
sql
-- Create database and user
CREATE DATABASE yemaya_gpu_jobs;
CREATE USER gpu_jobs_user WITH ENCRYPTED PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE yemaya_gpu_jobs TO gpu_jobs_user;
-- Enable extensions
\c yemaya_gpu_jobs
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";
bash
# Run migrations
pnpm --filter @yemaya/database db:migrate:deploy
Redis Setup#
bash
# redis.conf for production
maxmemory 2gb
maxmemory-policy allkeys-lru
appendonly yes
appendfsync everysec
tcp-keepalive 300
timeout 0
Object Storage Setup (MinIO)#
bash
# Create buckets
mc alias set yemaya http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb yemaya/gpu-jobs-inputs
mc mb yemaya/gpu-jobs-outputs
mc mb yemaya/gpu-jobs-artifacts
# Set lifecycle policies (auto-delete old files)
mc ilm add yemaya/gpu-jobs-artifacts --expiry-days 30
Configuration#
Production Configuration#
typescript
// config/production.ts
export const productionConfig: GPUJobsConfig = {
// API Configuration
api: {
host: '0.0.0.0',
port: 3000,
cors: {
origin: ['https://app.yemaya.example.com'],
credentials: true,
},
rateLimit: {
windowMs: 60000,
max: 100,
},
},
// Database
database: {
url: process.env.DATABASE_URL!,
pool: {
min: 5,
max: 20,
idleTimeoutMillis: 30000,
},
},
// Redis
redis: {
url: process.env.REDIS_URL!,
maxRetries: 3,
retryDelay: 1000,
},
// Providers
providers: {
'runpod-serverless': {
apiKey: process.env.RUNPOD_API_KEY!,
endpointId: process.env.RUNPOD_ENDPOINT_ID!,
timeout: 300000,
maxRetries: 3,
},
modal: {
token: process.env.MODAL_TOKEN!,
appName: 'yemaya-gpu-jobs',
},
},
// Job Processing
jobs: {
maxConcurrentPerUser: 5,
maxQueueDepth: 1000,
defaultTimeout: 3600,
retryConfig: {
maxRetries: 3,
initialDelayMs: 1000,
maxDelayMs: 30000,
backoffMultiplier: 2,
},
},
// Budget
budget: {
maxCostPerJob: 50.0,
dailyBudget: 500.0,
monthlyBudget: 5000.0,
alertWebhook: process.env.BUDGET_ALERT_WEBHOOK,
},
// Monitoring
monitoring: {
metricsPort: 9090,
healthCheckInterval: 30000,
alertWebhook: process.env.ALERT_WEBHOOK,
},
};
Environment Variables#
bash
# .env.production
NODE_ENV=production
LOG_LEVEL=info
# API
API_HOST=0.0.0.0
API_PORT=3000
API_SECRET=your_api_secret_key
# Database
DATABASE_URL=postgresql://gpu_jobs_user:password@postgres-primary.internal:5432/yemaya_gpu_jobs?sslmode=require
# Redis
REDIS_URL=redis://:password@redis-primary.internal:6379
# Storage
S3_ENDPOINT=https://s3.us-east-1.amazonaws.com
S3_ACCESS_KEY=AKIAXXXXXXXXXX
S3_SECRET_KEY=xxxxxxxxxxxxxxxx
S3_BUCKET=yemaya-gpu-jobs
S3_REGION=us-east-1
# GPU Providers
RUNPOD_API_KEY=xxxxxxxxxxxxxxxx
RUNPOD_ENDPOINT_ID=xxxxxxxx
MODAL_TOKEN=xxxxxxxxxxxxxxxx
# Monitoring
METRICS_PORT=9090
ALERT_WEBHOOK=https://hooks.slack.com/services/xxx/xxx/xxx
BUDGET_ALERT_WEBHOOK=https://hooks.slack.com/services/xxx/xxx/xxx
Scaling Configuration#
Horizontal Scaling#
yaml
# Kubernetes HPA Configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: gpu-jobs-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: gpu-jobs-api
minReplicas: 3
maxReplicas: 50
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 4
periodSeconds: 60
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: External
external:
metric:
name: gpu_jobs_queue_depth
target:
type: AverageValue
averageValue: '50'
Worker Scaling#
typescript
// Worker autoscaling configuration
const workerConfig = {
minWorkers: 2,
maxWorkers: 20,
scaleUpThreshold: 100, // Queue depth to trigger scale up
scaleDownThreshold: 10, // Queue depth to trigger scale down
scaleUpCooldown: 60000, // Cooldown after scale up (ms)
scaleDownCooldown: 300000, // Cooldown after scale down (ms)
};
Warm Pool Scaling#
typescript
// Predictive scaling for warm pools
const warmPoolConfig = {
pools: {
'texture-upscale': {
minWorkers: 2,
maxWorkers: 10,
targetUtilization: 0.7, // Scale up when 70% utilized
scaleUpIncrement: 2,
scaleDownIncrement: 1,
},
'blender-render': {
minWorkers: 1,
maxWorkers: 5,
targetUtilization: 0.8,
},
},
predictiveScaling: {
enabled: true,
lookAheadMinutes: 30,
historyDays: 7,
},
};
Monitoring Setup#
Prometheus Configuration#
yaml
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'gpu-jobs-api'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: gpu-jobs-api
action: keep
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: 'true'
action: keep
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
rule_files:
- /etc/prometheus/rules/*.yml
Alert Rules#
yaml
# prometheus-rules.yml
groups:
- name: gpu-jobs
rules:
- alert: GPUJobsHighFailureRate
expr: rate(gpu_jobs_failed_total[5m]) / rate(gpu_jobs_total[5m]) > 0.1
for: 5m
labels:
severity: critical
annotations:
summary: 'High GPU job failure rate'
description:
'GPU job failure rate is {{ $value | humanizePercentage }}'
- alert: GPUJobsQueueBacklog
expr: gpu_jobs_queue_depth > 500
for: 10m
labels:
severity: warning
annotations:
summary: 'GPU jobs queue backlog'
description: 'Queue depth is {{ $value }} jobs'
- alert: GPUJobsBudgetCritical
expr: gpu_jobs_budget_used_percent > 0.95
for: 1m
labels:
severity: critical
annotations:
summary: 'GPU jobs budget nearly exhausted'
description: 'Budget usage at {{ $value | humanizePercentage }}'
- alert: ProviderUnhealthy
expr: gpu_jobs_provider_healthy == 0
for: 5m
labels:
severity: warning
annotations:
summary: 'GPU provider unhealthy'
description: 'Provider {{ $labels.provider }} is unhealthy'
Grafana Dashboard#
json
{
"dashboard": {
"title": "GPU Jobs Overview",
"panels": [
{
"title": "Jobs Per Minute",
"type": "graph",
"targets": [
{
"expr": "rate(gpu_jobs_total[1m])",
"legendFormat": "{{ status }}"
}
]
},
{
"title": "Queue Depth",
"type": "gauge",
"targets": [
{
"expr": "gpu_jobs_queue_depth"
}
]
},
{
"title": "Cost per Hour",
"type": "stat",
"targets": [
{
"expr": "rate(gpu_jobs_cost_total[1h]) * 3600"
}
]
},
{
"title": "Provider Health",
"type": "table",
"targets": [
{
"expr": "gpu_jobs_provider_healthy",
"legendFormat": "{{ provider }}"
}
]
}
]
}
}
Security Considerations#
API Security#
typescript
// Security middleware configuration
const securityConfig = {
// Authentication
auth: {
jwtSecret: process.env.JWT_SECRET!,
tokenExpiration: '1h',
refreshTokenExpiration: '7d',
},
// Rate limiting per tier
rateLimits: {
free: { windowMs: 60000, max: 10 },
pro: { windowMs: 60000, max: 100 },
enterprise: { windowMs: 60000, max: 1000 },
},
// API key rotation
apiKeyRotation: {
enabled: true,
maxAge: 90, // days
warningDays: 14,
},
};
Secrets Management#
bash
# Using Kubernetes secrets
kubectl create secret generic gpu-jobs-secrets \
--from-literal=database-url='postgresql://...' \
--from-literal=redis-url='redis://...' \
--from-literal=runpod-api-key='...' \
--from-literal=modal-token='...'
# Using AWS Secrets Manager
aws secretsmanager create-secret \
--name gpu-jobs/database-url \
--secret-string 'postgresql://...'
Network Security#
yaml
# Kubernetes NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gpu-jobs-api
namespace: yemaya
spec:
podSelector:
matchLabels:
app: gpu-jobs-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- port: 3000
egress:
- to:
- namespaceSelector:
matchLabels:
name: yemaya
ports:
- port: 5432 # PostgreSQL
- port: 6379 # Redis
- to:
- namespaceSelector: {}
ports:
- port: 443 # External APIs (providers)
Disaster Recovery#
Backup Strategy#
bash
# Database backup (daily)
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
pg_dump $DATABASE_URL | gzip > /backups/gpu_jobs_$DATE.sql.gz
aws s3 cp /backups/gpu_jobs_$DATE.sql.gz s3://yemaya-backups/gpu-jobs/
# Keep last 30 days
find /backups -name "gpu_jobs_*.sql.gz" -mtime +30 -delete
Restore Procedure#
bash
# Restore from backup
gunzip -c gpu_jobs_20240115_120000.sql.gz | psql $DATABASE_URL
# Verify restoration
psql $DATABASE_URL -c "SELECT COUNT(*) FROM gpu_jobs;"
Failover Configuration#
typescript
// Multi-region failover
const failoverConfig = {
primary: {
region: 'us-east-1',
database: process.env.PRIMARY_DATABASE_URL,
redis: process.env.PRIMARY_REDIS_URL,
},
secondary: {
region: 'us-west-2',
database: process.env.SECONDARY_DATABASE_URL,
redis: process.env.SECONDARY_REDIS_URL,
},
healthCheckInterval: 30000,
failoverThreshold: 3, // Consecutive failures before failover
autoFailback: true,
failbackDelay: 300000, // 5 minutes
};
Operations Runbook#
Common Operations#
Restart API Service#
bash
# Kubernetes
kubectl rollout restart deployment/gpu-jobs-api -n yemaya
# Docker Compose
docker compose restart api
Scale Workers#
bash
# Scale up workers
kubectl scale deployment/gpu-jobs-workers --replicas=10 -n yemaya
# Scale down
kubectl scale deployment/gpu-jobs-workers --replicas=2 -n yemaya
Clear Job Queue#
bash
# Clear stuck jobs (use with caution)
redis-cli -u $REDIS_URL FLUSHDB
# Or selectively clear
redis-cli -u $REDIS_URL DEL "bull:gpu-jobs:waiting"
Check Provider Health#
bash
# API health check
curl https://api.yemaya.example.com/health
# Provider-specific health
curl https://api.yemaya.example.com/health/providers
Troubleshooting Procedures#
High Queue Depth#
- Check worker health:
kubectl get pods -l app=gpu-jobs-workers - Check provider status:
curl /health/providers - Scale workers if needed:
kubectl scale deployment... - Check for stuck jobs:
redis-cli LLEN bull:gpu-jobs:active
Provider Failures#
- Check provider dashboard (RunPod, Modal, AWS)
- Verify credentials are valid
- Check rate limits
- Enable failover to backup provider
Budget Alerts#
- Check current spend:
curl /api/v1/budget/status - Review expensive jobs:
curl /api/v1/jobs?sort=cost&limit=10 - Adjust budgets if needed
- Notify stakeholders