Eve library

Deployment Guide

Guide

Revision 1

On this page

Complete guide for deploying the Yemaya platform across different environments.

Table of Contents#

  1. Overview
  2. Prerequisites
  3. Local Development
  4. Staging Deployment
  5. Production Deployment
  6. Kubernetes Deployment
  7. Monitoring & Observability
  8. Scaling Strategies
  9. Backup & Recovery
  10. Security Hardening

Overview#

Architecture Components#

text
┌─────────────────────────────────────────────────────────────────┐
│                         Load Balancer                            │
│                         (Nginx/ALB)                              │
└─────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
        ┌───────────┐   ┌───────────┐   ┌───────────┐
        │  API (1)  │   │  API (2)  │   │  API (n)  │
        └─────┬─────┘   └─────┬─────┘   └─────┬─────┘
              │               │               │
              └───────────────┼───────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│  PostgreSQL   │   │     Redis     │   │     MinIO     │
│   (Primary)   │   │   (Cluster)   │   │   (Storage)   │
└───────────────┘   └───────────────┘   └───────────────┘
        │
        ▼
┌───────────────┐
│  PostgreSQL   │
│   (Replica)   │
└───────────────┘

        ┌───────────────────────────────────────────┐
        │              Worker Nodes                  │
        │  ┌─────────┐  ┌─────────┐  ┌─────────┐   │
        │  │ Agent   │  │ Media   │  │   3D    │   │
        │  │ Worker  │  │ Worker  │  │ Worker  │   │
        │  └─────────┘  └─────────┘  └─────────┘   │
        └───────────────────────────────────────────┘

Service Matrix#

Service Development Staging Production
API Server 1 instance 2 instances 2+ instances
Workers 1 instance 2 instances 3+ instances
PostgreSQL Single node Single + replica HA cluster
Redis Single node Single node Cluster
MinIO Single node Single node Distributed

Prerequisites#

Required Software#

bash
# Docker & Docker Compose
docker --version  # >= 24.0
docker compose version  # >= 2.20

# Node.js & pnpm
node --version  # >= 20.0
pnpm --version  # >= 10.0

# Optional: Kubernetes tools
kubectl version --client  # >= 1.28
helm version  # >= 3.12

Hardware Requirements#

Environment CPU RAM Storage
Development 4 cores 8 GB 50 GB SSD
Staging 8 cores 16 GB 200 GB SSD
Production 16+ cores 32+ GB 500+ GB SSD

Network Requirements#

Port Service Access
80/443 HTTP/HTTPS Public
3000 API Server Internal/Load Balancer
5432 PostgreSQL Internal only
6379 Redis Internal only
9000/9001 MinIO Internal only

Local Development#

Quick Start#

bash
# 1. Clone repository
git clone https://github.com/yemaya/yemaya.git
cd yemaya

# 2. Copy environment file
cp .env.example .env

# 3. Start infrastructure
docker compose up -d

# 4. Install dependencies
pnpm install

# 5. Run database migrations
pnpm --filter @yemaya/database db:migrate

# 6. Start development servers
pnpm dev

Environment Configuration#

Edit .env with your local settings:

bash
# Database
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya

# Redis
REDIS_URL=redis://localhost:6379

# MinIO
MINIO_ENDPOINT=localhost
MINIO_PORT=9000
MINIO_ACCESS_KEY=yemaya_minio
MINIO_SECRET_KEY=yemaya_minio_secret

# Authentication
JWT_SECRET=development-jwt-secret-change-in-production-32chars

# AI Providers (optional)
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here

Docker Compose Profiles#

bash
# Core services only
docker compose up -d

# With Vault (secret management)
docker compose --profile vault up -d

# With ELK Stack (logging)
docker compose --profile logging up -d

# With Prometheus/Grafana (monitoring)
docker compose --profile monitoring up -d

# With backup service
docker compose --profile backup up -d

# All services
docker compose --profile vault --profile logging --profile monitoring up -d

Available Services#

Service URL Credentials
API Server http://localhost:3000 -
API Docs (Swagger) http://localhost:3000/docs -
API Reference (Scalar) http://localhost:3000/reference -
MinIO Console http://localhost:9001 yemaya_minio / yemaya_minio_secret
Mailpit (Email) http://localhost:8025 -
Vault UI http://localhost:8200 Token: yemaya-dev-token
Kibana http://localhost:5601 elastic / yemaya_elastic_dev
Grafana http://localhost:3001 admin / yemaya_grafana_dev
Prometheus http://localhost:9090 -

Development Commands#

bash
# Run specific app
pnpm --filter @yemaya/api dev
pnpm --filter @yemaya/web dev
pnpm --filter @yemaya/workers dev

# Run tests
pnpm test
pnpm --filter @yemaya/core test:watch

# Database operations
pnpm --filter @yemaya/database db:migrate
pnpm --filter @yemaya/database db:studio
pnpm --filter @yemaya/database db:generate

# Build all packages
pnpm build

# Type checking
pnpm typecheck

# Linting
pnpm lint
pnpm lint:fix

Staging Deployment#

Infrastructure Setup#

  1. Provision servers (2 app servers, 1 database server, 1 storage server)

  2. Configure environment variables:

bash
# /etc/yemaya/.env.staging

NODE_ENV=staging

# Database
POSTGRES_USER=yemaya_staging
POSTGRES_PASSWORD=<generated-password>
POSTGRES_DB=yemaya_staging
DATABASE_URL=postgresql://yemaya_staging:<password>@db-staging:5432/yemaya_staging

# Redis
REDIS_PASSWORD=<generated-password>
REDIS_URL=redis://:<password>@redis-staging:6379

# MinIO
MINIO_ACCESS_KEY=<generated-key>
MINIO_SECRET_KEY=<generated-secret>

# JWT
JWT_SECRET=<generated-secret-32-chars>

# AI Keys
ANTHROPIC_API_KEY=sk-ant-<staging-key>
OPENAI_API_KEY=sk-<staging-key>
  1. Deploy with Docker Compose:
bash
# On staging server
docker compose -f docker-compose.prod.yml --env-file .env.staging up -d

Continuous Deployment#

yaml
# .github/workflows/deploy-staging.yml
name: Deploy to Staging

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build Docker images
        run: |
          docker compose -f docker-compose.prod.yml build

      - name: Push to registry
        run: |
          docker tag yemaya-api:latest ${{ secrets.REGISTRY }}/yemaya-api:staging
          docker push ${{ secrets.REGISTRY }}/yemaya-api:staging

      - name: Deploy to staging
        uses: appleboy/ssh-action@v1.0.0
        with:
          host: ${{ secrets.STAGING_HOST }}
          username: deploy
          key: ${{ secrets.DEPLOY_KEY }}
          script: |
            cd /opt/yemaya
            docker compose pull
            docker compose -f docker-compose.prod.yml up -d
            docker compose exec api pnpm db:migrate

Production Deployment#

Pre-Deployment Checklist#

  • All environment variables set
  • SSL certificates obtained (Let's Encrypt or custom)
  • Database backups configured
  • Monitoring and alerting configured
  • Rate limiting configured
  • Security headers configured
  • CDN configured for static assets

Infrastructure Requirements#

yaml
# Minimum production infrastructure

API Servers (2+):
  - 4 vCPU, 8 GB RAM
  - Auto-scaling group

Worker Servers (3+):
  - 8 vCPU, 16 GB RAM (for media processing)
  - 4 vCPU, 8 GB RAM (for general workers)

Database:
  - PostgreSQL 16
  - 8 vCPU, 32 GB RAM
  - 500 GB SSD (gp3)
  - Multi-AZ deployment

Cache:
  - Redis 7
  - 2 vCPU, 8 GB RAM
  - Cluster mode enabled

Storage:
  - MinIO or AWS S3
  - 1 TB initial
  - Versioning enabled
  - Lifecycle policies

Environment Variables#

bash
# /etc/yemaya/.env.production

NODE_ENV=production

# Database (use connection pooler)
DATABASE_URL=postgresql://yemaya:<password>@pgbouncer:6432/yemaya?pgbouncer=true
DATABASE_URL_DIRECT=postgresql://yemaya:<password>@db-primary:5432/yemaya

# Redis (cluster mode)
REDIS_URL=redis://:<password>@redis-cluster:6379

# MinIO (distributed mode) or S3
MINIO_ENDPOINT=storage.yemaya.io
MINIO_PORT=443
MINIO_USE_SSL=true
MINIO_ACCESS_KEY=<production-key>
MINIO_SECRET_KEY=<production-secret>

# Authentication
JWT_SECRET=<production-secret-64-chars>
JWT_EXPIRES_IN=15m
JWT_REFRESH_EXPIRES_IN=7d

# AI Providers
ANTHROPIC_API_KEY=sk-ant-<production-key>
OPENAI_API_KEY=sk-<production-key>

# Email (production SMTP)
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASS=<sendgrid-api-key>
SMTP_FROM=noreply@yemaya.io

# Monitoring
SENTRY_DSN=https://<key>@sentry.io/<project>

Deployment Steps#

bash
# 1. Build production images
docker compose -f docker-compose.prod.yml build

# 2. Push to container registry
docker tag yemaya-api:latest registry.yemaya.io/yemaya-api:v1.0.0
docker push registry.yemaya.io/yemaya-api:v1.0.0

# 3. Deploy database migrations (from bastion host)
docker compose exec api pnpm db:migrate

# 4. Rolling update (zero-downtime)
docker compose -f docker-compose.prod.yml up -d --no-deps --scale api=2

# 5. Health check
curl https://api.yemaya.io/health

Nginx Configuration#

nginx
# /etc/nginx/sites-available/yemaya.conf

upstream yemaya_api {
    least_conn;
    server api-1:3000 weight=1 max_fails=3 fail_timeout=30s;
    server api-2:3000 weight=1 max_fails=3 fail_timeout=30s;
    keepalive 32;
}

server {
    listen 80;
    server_name api.yemaya.io;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name api.yemaya.io;

    ssl_certificate /etc/letsencrypt/live/api.yemaya.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/api.yemaya.io/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    # Security headers
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header Content-Security-Policy "default-src 'self'" always;

    # Gzip compression
    gzip on;
    gzip_types text/plain application/json application/javascript text/css;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;
    limit_req zone=api burst=200 nodelay;

    location / {
        proxy_pass http://yemaya_api;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    location /health {
        proxy_pass http://yemaya_api;
        access_log off;
    }

    # WebSocket endpoint
    location /ws {
        proxy_pass http://yemaya_api;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
    }
}

Kubernetes Deployment#

Helm Chart Structure#

text
charts/yemaya/
├── Chart.yaml
├── values.yaml
├── values-staging.yaml
├── values-production.yaml
├── templates/
│   ├── _helpers.tpl
│   ├── configmap.yaml
│   ├── secret.yaml
│   ├── deployment-api.yaml
│   ├── deployment-workers.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   └── pdb.yaml

values.yaml#

yaml
# charts/yemaya/values.yaml

replicaCount:
  api: 2
  workers: 3

image:
  repository: registry.yemaya.io/yemaya
  pullPolicy: IfNotPresent
  tag: 'latest'

api:
  resources:
    limits:
      cpu: 2000m
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 1Gi

workers:
  resources:
    limits:
      cpu: 4000m
      memory: 8Gi
    requests:
      cpu: 1000m
      memory: 2Gi

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rate-limit: '100'
  hosts:
    - host: api.yemaya.io
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: yemaya-tls
      hosts:
        - api.yemaya.io

postgresql:
  enabled: true
  auth:
    postgresPassword: ''
    username: yemaya
    password: ''
    database: yemaya
  primary:
    persistence:
      size: 100Gi
  readReplicas:
    replicaCount: 1

redis:
  enabled: true
  auth:
    password: ''
  master:
    persistence:
      size: 10Gi

minio:
  enabled: true
  auth:
    rootUser: ''
    rootPassword: ''
  persistence:
    size: 500Gi

Deployment Commands#

bash
# Add Helm repositories
helm repo add bitnami https://charts.bitnami.com/bitnami

# Create namespace
kubectl create namespace yemaya

# Create secrets
kubectl create secret generic yemaya-secrets \
  --from-literal=jwt-secret=<secret> \
  --from-literal=anthropic-api-key=<key> \
  -n yemaya

# Install with Helm
helm install yemaya ./charts/yemaya \
  -f ./charts/yemaya/values-production.yaml \
  -n yemaya

# Upgrade deployment
helm upgrade yemaya ./charts/yemaya \
  -f ./charts/yemaya/values-production.yaml \
  -n yemaya

# Check status
kubectl get pods -n yemaya
kubectl get ingress -n yemaya

Horizontal Pod Autoscaler#

yaml
# templates/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "yemaya.fullname" . }}-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "yemaya.fullname" . }}-api
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15

Monitoring & Observability#

ELK Stack Setup#

The development docker-compose includes full ELK stack:

bash
# Start with logging profile
docker compose --profile logging up -d

# Access Kibana
open http://localhost:5601
# Login: elastic / yemaya_elastic_dev

Log Configuration#

Application logs are structured JSON:

json
{
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "level": "info",
  "service": "api",
  "environment": "production",
  "message": "Request completed",
  "correlation_id": "req-abc123",
  "user_id": "usr-xyz789",
  "http": {
    "method": "GET",
    "path": "/v1/projects",
    "status_code": 200
  },
  "duration_ms": 45
}

Prometheus Metrics#

Exposed at /metrics:

text
# API metrics
http_requests_total{method="GET", path="/v1/projects", status="200"}
http_request_duration_seconds_bucket{le="0.1"}
http_active_connections

# Worker metrics
worker_jobs_processed_total{queue="agent", status="completed"}
worker_jobs_duration_seconds_bucket{le="10"}
worker_active_jobs{queue="media"}

# Database metrics
db_connections_active
db_queries_total
db_query_duration_seconds

# Cache metrics
redis_operations_total{operation="get"}
redis_hit_ratio

Grafana Dashboards#

Pre-configured dashboards available:

  1. API Overview - Request rates, latencies, error rates
  2. Worker Health - Job processing, queue depths
  3. Database Performance - Connections, query times
  4. Infrastructure - CPU, memory, disk usage

Alerting Rules#

yaml
# infrastructure/prometheus/rules/alerts.yml

groups:
  - name: yemaya-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High error rate detected
          description: Error rate is {{ $value }}

      - alert: SlowResponses
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Slow response times
          description: 95th percentile latency is {{ $value }}s

      - alert: WorkerQueueBacklog
        expr: worker_queue_depth > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: Worker queue backlog growing
          description: Queue {{ $labels.queue }} has {{ $value }} pending jobs

      - alert: DatabaseConnectionsHigh
        expr: db_connections_active / db_connections_max > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Database connections running high
          description: {{ $value | humanizePercentage }} of max connections used

Scaling Strategies#

Horizontal Scaling#

yaml
# API scaling based on CPU/memory
api:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 20
    targetCPU: 70
    targetMemory: 80

# Worker scaling based on queue depth
workers:
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 50
    metrics:
      - type: External
        external:
          metric:
            name: worker_queue_depth
          target:
            type: Value
            value: 100

Database Scaling#

  1. Read Replicas: Route read queries to replicas
  2. Connection Pooling: Use PgBouncer for connection management
  3. Partitioning: Partition large tables by date
yaml
# PgBouncer configuration
pgbouncer:
  pool_mode: transaction
  max_client_conn: 1000
  default_pool_size: 20
  reserve_pool_size: 5
  reserve_pool_timeout: 3

Cache Scaling#

yaml
# Redis Cluster configuration
redis:
  cluster:
    enabled: true
    nodes: 6
    replicas: 1
  master:
    resources:
      limits:
        memory: 4Gi

Backup & Recovery#

Database Backups#

bash
# Automated daily backups
docker compose run --rm backup pg_dump -Fc > /backups/yemaya-$(date +%Y%m%d).dump

# Restore from backup
docker compose exec postgres pg_restore -d yemaya /backups/yemaya-20240115.dump

Backup Configuration#

yaml
# infrastructure/backup/config.yml

schedule:
  full: '0 2 * * *' # Daily at 2 AM
  incremental: '0 * * * *' # Hourly

retention:
  days: 30
  weekly: 12
  monthly: 12

storage:
  type: s3
  bucket: yemaya-backups
  encryption: AES256

notifications:
  slack:
    webhook: ${SLACK_WEBHOOK}
    on_failure: true
    on_success: false

Disaster Recovery#

  1. RTO (Recovery Time Objective): 1 hour
  2. RPO (Recovery Point Objective): 1 hour
bash
# Disaster recovery procedure
1. Restore database from latest backup
2. Deploy application to new infrastructure
3. Update DNS to point to new endpoints
4. Verify all services healthy
5. Notify stakeholders

Security Hardening#

Secret Management#

Use HashiCorp Vault for production secrets:

bash
# Enable Vault
docker compose --profile vault up -d

# Store secrets
vault kv put secret/yemaya/production \
  jwt_secret=<secret> \
  anthropic_api_key=<key> \
  database_password=<password>

# Application reads from Vault
VAULT_ADDR=https://vault.yemaya.io:8200
VAULT_TOKEN=<app-token>

Network Security#

yaml
# Network policies (Kubernetes)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: yemaya-network-policy
spec:
  podSelector:
    matchLabels:
      app: yemaya-api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - protocol: TCP
          port: 3000
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgresql
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379

Security Checklist#

  • All secrets stored in Vault or K8s secrets
  • TLS 1.3 enabled on all endpoints
  • Rate limiting configured
  • CORS properly configured
  • SQL injection prevention (parameterized queries)
  • XSS prevention (Content-Security-Policy)
  • CSRF tokens for mutations
  • Audit logging enabled
  • Container images scanned
  • Dependency vulnerabilities checked

Troubleshooting#

See Troubleshooting Guide for common issues and solutions.


Quick Reference#

Common Commands#

bash
# View logs
docker compose logs -f api
docker compose logs -f workers

# Restart service
docker compose restart api

# Scale workers
docker compose up -d --scale workers=5

# Database shell
docker compose exec postgres psql -U yemaya -d yemaya

# Redis CLI
docker compose exec redis redis-cli

# Run migrations
docker compose exec api pnpm db:migrate

# Health check
curl http://localhost:3000/health
curl http://localhost:3000/health/ready

Environment URLs#

Environment API URL Docs URL
Development http://localhost:3000 http://localhost:3000/docs
Staging https://api.staging.yemaya.io https://api.staging.yemaya.io/docs
Production https://api.yemaya.io https://api.yemaya.io/docs

See Also#