# GPU Jobs Deployment Guide

This guide covers deploying the GPU Jobs system in production environments.

## Table of Contents

- [Architecture Overview](#architecture-overview)
- [Prerequisites](#prerequisites)
- [Deployment Options](#deployment-options)
- [Infrastructure Setup](#infrastructure-setup)
- [Configuration](#configuration)
- [Scaling Configuration](#scaling-configuration)
- [Monitoring Setup](#monitoring-setup)
- [Security Considerations](#security-considerations)
- [Disaster Recovery](#disaster-recovery)
- [Operations Runbook](#operations-runbook)

## Architecture Overview

```
                           ┌─────────────────────────────────────────────────────┐
                           │                  Load Balancer                       │
                           └────────────────────────┬────────────────────────────┘
                                                    │
                    ┌───────────────────────────────┼───────────────────────────────┐
                    │                               │                               │
            ┌───────▼───────┐               ┌───────▼───────┐               ┌───────▼───────┐
            │   API Server  │               │   API Server  │               │   API Server  │
            │   (Node.js)   │               │   (Node.js)   │               │   (Node.js)   │
            └───────┬───────┘               └───────┬───────┘               └───────┬───────┘
                    │                               │                               │
                    └───────────────────────────────┼───────────────────────────────┘
                                                    │
    ┌──────────────────┬────────────────────────────┼────────────────────────────┬──────────────────┐
    │                  │                            │                            │                  │
┌───▼───┐        ┌─────▼─────┐              ┌───────▼───────┐              ┌──────▼──────┐    ┌─────▼─────┐
│ Redis │        │ PostgreSQL│              │  Job Queue    │              │  Object     │    │ Monitoring│
│ Cache │        │  Database │              │  (BullMQ)     │              │  Storage    │    │ (Grafana) │
└───────┘        └───────────┘              └───────┬───────┘              │  (S3/MinIO) │    └───────────┘
                                                    │                      └─────────────┘
                    ┌───────────────────────────────┼───────────────────────────────┐
                    │                               │                               │
            ┌───────▼───────┐               ┌───────▼───────┐               ┌───────▼───────┐
            │ GPU Provider  │               │ GPU Provider  │               │ GPU Provider  │
            │   (RunPod)    │               │   (Modal)     │               │   (AWS ECS)   │
            └───────────────┘               └───────────────┘               └───────────────┘
```

## Prerequisites

### Required Services

- **PostgreSQL 15+**: Job state persistence
- **Redis 7+**: Caching and job queue
- **S3-compatible storage**: MinIO, AWS S3, or GCS
- **Container runtime**: Docker or Kubernetes

### Required Credentials

```bash
# Database
DATABASE_URL=postgresql://user:password@host:5432/yemaya_gpu_jobs

# Cache
REDIS_URL=redis://:password@host:6379

# Storage
S3_ENDPOINT=https://s3.amazonaws.com
S3_ACCESS_KEY=your_access_key
S3_SECRET_KEY=your_secret_key
S3_BUCKET=yemaya-gpu-jobs

# GPU Providers (at least one required)
RUNPOD_API_KEY=your_runpod_key
RUNPOD_ENDPOINT_ID=your_endpoint_id
MODAL_TOKEN=your_modal_token
AWS_ACCESS_KEY_ID=your_aws_key
AWS_SECRET_ACCESS_KEY=your_aws_secret
```

## Deployment Options

### Option 1: Docker Compose (Small Scale)

```yaml
# docker-compose.prod.yml
version: '3.8'
services:
  api:
    image: your-registry/yemaya-api:latest
    environment:
      - NODE_ENV=production
      - DATABASE_URL=${DATABASE_URL}
      - REDIS_URL=${REDIS_URL}
      - RUNPOD_API_KEY=${RUNPOD_API_KEY}
    ports:
      - '3000:3000'
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '2'
          memory: 4G
    healthcheck:
      test: ['CMD', 'curl', '-f', 'http://localhost:3000/health']
      interval: 30s
      timeout: 10s
      retries: 3

  workers:
    image: your-registry/yemaya-workers:latest
    environment:
      - NODE_ENV=production
      - DATABASE_URL=${DATABASE_URL}
      - REDIS_URL=${REDIS_URL}
      - RUNPOD_API_KEY=${RUNPOD_API_KEY}
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: '1'
          memory: 2G

  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: yemaya_gpu_jobs
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 4G

  redis:
    image: redis:7-alpine
    command: redis-server --requirepass ${REDIS_PASSWORD}
    volumes:
      - redis_data:/data
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 2G

volumes:
  postgres_data:
  redis_data:
```

Deploy:

```bash
docker compose -f docker-compose.prod.yml up -d
```

### Option 2: Kubernetes (Production Scale)

```yaml
# kubernetes/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-jobs-api
  namespace: yemaya
spec:
  replicas: 3
  selector:
    matchLabels:
      app: gpu-jobs-api
  template:
    metadata:
      labels:
        app: gpu-jobs-api
    spec:
      containers:
        - name: api
          image: your-registry/yemaya-api:latest
          ports:
            - containerPort: 3000
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: gpu-jobs-secrets
                  key: database-url
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: gpu-jobs-secrets
                  key: redis-url
            - name: RUNPOD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: gpu-jobs-secrets
                  key: runpod-api-key
          resources:
            requests:
              cpu: '500m'
              memory: '1Gi'
            limits:
              cpu: '2000m'
              memory: '4Gi'
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: gpu-jobs-api
  namespace: yemaya
spec:
  selector:
    app: gpu-jobs-api
  ports:
    - port: 80
      targetPort: 3000
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gpu-jobs-api
  namespace: yemaya
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - api.yemaya.example.com
      secretName: gpu-jobs-tls
  rules:
    - host: api.yemaya.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gpu-jobs-api
                port:
                  number: 80
```

```yaml
# kubernetes/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-jobs-api
  namespace: yemaya
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gpu-jobs-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
```

Deploy:

```bash
kubectl apply -f kubernetes/
```

### Option 3: AWS ECS Fargate

```json
{
  "family": "gpu-jobs-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "executionRoleArn": "arn:aws:iam::ACCOUNT:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::ACCOUNT:role/gpuJobsTaskRole",
  "containerDefinitions": [
    {
      "name": "api",
      "image": "ACCOUNT.dkr.ecr.REGION.amazonaws.com/yemaya-api:latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "environment": [{ "name": "NODE_ENV", "value": "production" }],
      "secrets": [
        {
          "name": "DATABASE_URL",
          "valueFrom": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:gpu-jobs/database-url"
        },
        {
          "name": "RUNPOD_API_KEY",
          "valueFrom": "arn:aws:secretsmanager:REGION:ACCOUNT:secret:gpu-jobs/runpod-key"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/gpu-jobs-api",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost:3000/health || exit 1"
        ],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    }
  ]
}
```

## Infrastructure Setup

### Database Setup

```sql
-- Create database and user
CREATE DATABASE yemaya_gpu_jobs;
CREATE USER gpu_jobs_user WITH ENCRYPTED PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE yemaya_gpu_jobs TO gpu_jobs_user;

-- Enable extensions
\c yemaya_gpu_jobs
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";
```

```bash
# Run migrations
pnpm --filter @yemaya/database db:migrate:deploy
```

### Redis Setup

```bash
# redis.conf for production
maxmemory 2gb
maxmemory-policy allkeys-lru
appendonly yes
appendfsync everysec
tcp-keepalive 300
timeout 0
```

### Object Storage Setup (MinIO)

```bash
# Create buckets
mc alias set yemaya http://minio:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb yemaya/gpu-jobs-inputs
mc mb yemaya/gpu-jobs-outputs
mc mb yemaya/gpu-jobs-artifacts

# Set lifecycle policies (auto-delete old files)
mc ilm add yemaya/gpu-jobs-artifacts --expiry-days 30
```

## Configuration

### Production Configuration

```typescript
// config/production.ts
export const productionConfig: GPUJobsConfig = {
  // API Configuration
  api: {
    host: '0.0.0.0',
    port: 3000,
    cors: {
      origin: ['https://app.yemaya.example.com'],
      credentials: true,
    },
    rateLimit: {
      windowMs: 60000,
      max: 100,
    },
  },

  // Database
  database: {
    url: process.env.DATABASE_URL!,
    pool: {
      min: 5,
      max: 20,
      idleTimeoutMillis: 30000,
    },
  },

  // Redis
  redis: {
    url: process.env.REDIS_URL!,
    maxRetries: 3,
    retryDelay: 1000,
  },

  // Providers
  providers: {
    'runpod-serverless': {
      apiKey: process.env.RUNPOD_API_KEY!,
      endpointId: process.env.RUNPOD_ENDPOINT_ID!,
      timeout: 300000,
      maxRetries: 3,
    },
    modal: {
      token: process.env.MODAL_TOKEN!,
      appName: 'yemaya-gpu-jobs',
    },
  },

  // Job Processing
  jobs: {
    maxConcurrentPerUser: 5,
    maxQueueDepth: 1000,
    defaultTimeout: 3600,
    retryConfig: {
      maxRetries: 3,
      initialDelayMs: 1000,
      maxDelayMs: 30000,
      backoffMultiplier: 2,
    },
  },

  // Budget
  budget: {
    maxCostPerJob: 50.0,
    dailyBudget: 500.0,
    monthlyBudget: 5000.0,
    alertWebhook: process.env.BUDGET_ALERT_WEBHOOK,
  },

  // Monitoring
  monitoring: {
    metricsPort: 9090,
    healthCheckInterval: 30000,
    alertWebhook: process.env.ALERT_WEBHOOK,
  },
};
```

### Environment Variables

```bash
# .env.production
NODE_ENV=production
LOG_LEVEL=info

# API
API_HOST=0.0.0.0
API_PORT=3000
API_SECRET=your_api_secret_key

# Database
DATABASE_URL=postgresql://gpu_jobs_user:password@postgres-primary.internal:5432/yemaya_gpu_jobs?sslmode=require

# Redis
REDIS_URL=redis://:password@redis-primary.internal:6379

# Storage
S3_ENDPOINT=https://s3.us-east-1.amazonaws.com
S3_ACCESS_KEY=AKIAXXXXXXXXXX
S3_SECRET_KEY=xxxxxxxxxxxxxxxx
S3_BUCKET=yemaya-gpu-jobs
S3_REGION=us-east-1

# GPU Providers
RUNPOD_API_KEY=xxxxxxxxxxxxxxxx
RUNPOD_ENDPOINT_ID=xxxxxxxx
MODAL_TOKEN=xxxxxxxxxxxxxxxx

# Monitoring
METRICS_PORT=9090
ALERT_WEBHOOK=https://hooks.slack.com/services/xxx/xxx/xxx
BUDGET_ALERT_WEBHOOK=https://hooks.slack.com/services/xxx/xxx/xxx
```

## Scaling Configuration

### Horizontal Scaling

```yaml
# Kubernetes HPA Configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-jobs-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gpu-jobs-api
  minReplicas: 3
  maxReplicas: 50
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: External
      external:
        metric:
          name: gpu_jobs_queue_depth
        target:
          type: AverageValue
          averageValue: '50'
```

### Worker Scaling

```typescript
// Worker autoscaling configuration
const workerConfig = {
  minWorkers: 2,
  maxWorkers: 20,
  scaleUpThreshold: 100, // Queue depth to trigger scale up
  scaleDownThreshold: 10, // Queue depth to trigger scale down
  scaleUpCooldown: 60000, // Cooldown after scale up (ms)
  scaleDownCooldown: 300000, // Cooldown after scale down (ms)
};
```

### Warm Pool Scaling

```typescript
// Predictive scaling for warm pools
const warmPoolConfig = {
  pools: {
    'texture-upscale': {
      minWorkers: 2,
      maxWorkers: 10,
      targetUtilization: 0.7, // Scale up when 70% utilized
      scaleUpIncrement: 2,
      scaleDownIncrement: 1,
    },
    'blender-render': {
      minWorkers: 1,
      maxWorkers: 5,
      targetUtilization: 0.8,
    },
  },
  predictiveScaling: {
    enabled: true,
    lookAheadMinutes: 30,
    historyDays: 7,
  },
};
```

## Monitoring Setup

### Prometheus Configuration

```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'gpu-jobs-api'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: gpu-jobs-api
        action: keep
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        regex: 'true'
        action: keep

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/rules/*.yml
```

### Alert Rules

```yaml
# prometheus-rules.yml
groups:
  - name: gpu-jobs
    rules:
      - alert: GPUJobsHighFailureRate
        expr: rate(gpu_jobs_failed_total[5m]) / rate(gpu_jobs_total[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'High GPU job failure rate'
          description:
            'GPU job failure rate is {{ $value | humanizePercentage }}'

      - alert: GPUJobsQueueBacklog
        expr: gpu_jobs_queue_depth > 500
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'GPU jobs queue backlog'
          description: 'Queue depth is {{ $value }} jobs'

      - alert: GPUJobsBudgetCritical
        expr: gpu_jobs_budget_used_percent > 0.95
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'GPU jobs budget nearly exhausted'
          description: 'Budget usage at {{ $value | humanizePercentage }}'

      - alert: ProviderUnhealthy
        expr: gpu_jobs_provider_healthy == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'GPU provider unhealthy'
          description: 'Provider {{ $labels.provider }} is unhealthy'
```

### Grafana Dashboard

```json
{
  "dashboard": {
    "title": "GPU Jobs Overview",
    "panels": [
      {
        "title": "Jobs Per Minute",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(gpu_jobs_total[1m])",
            "legendFormat": "{{ status }}"
          }
        ]
      },
      {
        "title": "Queue Depth",
        "type": "gauge",
        "targets": [
          {
            "expr": "gpu_jobs_queue_depth"
          }
        ]
      },
      {
        "title": "Cost per Hour",
        "type": "stat",
        "targets": [
          {
            "expr": "rate(gpu_jobs_cost_total[1h]) * 3600"
          }
        ]
      },
      {
        "title": "Provider Health",
        "type": "table",
        "targets": [
          {
            "expr": "gpu_jobs_provider_healthy",
            "legendFormat": "{{ provider }}"
          }
        ]
      }
    ]
  }
}
```

## Security Considerations

### API Security

```typescript
// Security middleware configuration
const securityConfig = {
  // Authentication
  auth: {
    jwtSecret: process.env.JWT_SECRET!,
    tokenExpiration: '1h',
    refreshTokenExpiration: '7d',
  },

  // Rate limiting per tier
  rateLimits: {
    free: { windowMs: 60000, max: 10 },
    pro: { windowMs: 60000, max: 100 },
    enterprise: { windowMs: 60000, max: 1000 },
  },

  // API key rotation
  apiKeyRotation: {
    enabled: true,
    maxAge: 90, // days
    warningDays: 14,
  },
};
```

### Secrets Management

```bash
# Using Kubernetes secrets
kubectl create secret generic gpu-jobs-secrets \
  --from-literal=database-url='postgresql://...' \
  --from-literal=redis-url='redis://...' \
  --from-literal=runpod-api-key='...' \
  --from-literal=modal-token='...'

# Using AWS Secrets Manager
aws secretsmanager create-secret \
  --name gpu-jobs/database-url \
  --secret-string 'postgresql://...'
```

### Network Security

```yaml
# Kubernetes NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gpu-jobs-api
  namespace: yemaya
spec:
  podSelector:
    matchLabels:
      app: gpu-jobs-api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - port: 3000
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: yemaya
      ports:
        - port: 5432 # PostgreSQL
        - port: 6379 # Redis
    - to:
        - namespaceSelector: {}
      ports:
        - port: 443 # External APIs (providers)
```

## Disaster Recovery

### Backup Strategy

```bash
# Database backup (daily)
#!/bin/bash
DATE=$(date +%Y%m%d_%H%M%S)
pg_dump $DATABASE_URL | gzip > /backups/gpu_jobs_$DATE.sql.gz
aws s3 cp /backups/gpu_jobs_$DATE.sql.gz s3://yemaya-backups/gpu-jobs/

# Keep last 30 days
find /backups -name "gpu_jobs_*.sql.gz" -mtime +30 -delete
```

### Restore Procedure

```bash
# Restore from backup
gunzip -c gpu_jobs_20240115_120000.sql.gz | psql $DATABASE_URL

# Verify restoration
psql $DATABASE_URL -c "SELECT COUNT(*) FROM gpu_jobs;"
```

### Failover Configuration

```typescript
// Multi-region failover
const failoverConfig = {
  primary: {
    region: 'us-east-1',
    database: process.env.PRIMARY_DATABASE_URL,
    redis: process.env.PRIMARY_REDIS_URL,
  },
  secondary: {
    region: 'us-west-2',
    database: process.env.SECONDARY_DATABASE_URL,
    redis: process.env.SECONDARY_REDIS_URL,
  },
  healthCheckInterval: 30000,
  failoverThreshold: 3, // Consecutive failures before failover
  autoFailback: true,
  failbackDelay: 300000, // 5 minutes
};
```

## Operations Runbook

### Common Operations

#### Restart API Service

```bash
# Kubernetes
kubectl rollout restart deployment/gpu-jobs-api -n yemaya

# Docker Compose
docker compose restart api
```

#### Scale Workers

```bash
# Scale up workers
kubectl scale deployment/gpu-jobs-workers --replicas=10 -n yemaya

# Scale down
kubectl scale deployment/gpu-jobs-workers --replicas=2 -n yemaya
```

#### Clear Job Queue

```bash
# Clear stuck jobs (use with caution)
redis-cli -u $REDIS_URL FLUSHDB

# Or selectively clear
redis-cli -u $REDIS_URL DEL "bull:gpu-jobs:waiting"
```

#### Check Provider Health

```bash
# API health check
curl https://api.yemaya.example.com/health

# Provider-specific health
curl https://api.yemaya.example.com/health/providers
```

### Troubleshooting Procedures

#### High Queue Depth

1. Check worker health: `kubectl get pods -l app=gpu-jobs-workers`
2. Check provider status: `curl /health/providers`
3. Scale workers if needed: `kubectl scale deployment...`
4. Check for stuck jobs: `redis-cli LLEN bull:gpu-jobs:active`

#### Provider Failures

1. Check provider dashboard (RunPod, Modal, AWS)
2. Verify credentials are valid
3. Check rate limits
4. Enable failover to backup provider

#### Budget Alerts

1. Check current spend: `curl /api/v1/budget/status`
2. Review expensive jobs: `curl /api/v1/jobs?sort=cost&limit=10`
3. Adjust budgets if needed
4. Notify stakeholders

## Related Documentation

- [GPU Jobs README](./README.md)
- [Provider Configuration](./providers.md)
- [Cost Model](./cost-model.md)
- [Troubleshooting](./troubleshooting.md)
- [Local Development](./local-development.md)
