Complete guide for deploying the Yemaya platform across different environments.
Table of Contents#
- Overview
- Prerequisites
- Local Development
- Staging Deployment
- Production Deployment
- Kubernetes Deployment
- Monitoring & Observability
- Scaling Strategies
- Backup & Recovery
- Security Hardening
Overview#
Architecture Components#
text
┌─────────────────────────────────────────────────────────────────┐
│ Load Balancer │
│ (Nginx/ALB) │
└─────────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ API (1) │ │ API (2) │ │ API (n) │
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
│ │ │
└───────────────┼───────────────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ PostgreSQL │ │ Redis │ │ MinIO │
│ (Primary) │ │ (Cluster) │ │ (Storage) │
└───────────────┘ └───────────────┘ └───────────────┘
│
▼
┌───────────────┐
│ PostgreSQL │
│ (Replica) │
└───────────────┘
┌───────────────────────────────────────────┐
│ Worker Nodes │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Agent │ │ Media │ │ 3D │ │
│ │ Worker │ │ Worker │ │ Worker │ │
│ └─────────┘ └─────────┘ └─────────┘ │
└───────────────────────────────────────────┘
Service Matrix#
| Service | Development | Staging | Production |
|---|---|---|---|
| API Server | 1 instance | 2 instances | 2+ instances |
| Workers | 1 instance | 2 instances | 3+ instances |
| PostgreSQL | Single node | Single + replica | HA cluster |
| Redis | Single node | Single node | Cluster |
| MinIO | Single node | Single node | Distributed |
Prerequisites#
Required Software#
bash
# Docker & Docker Compose
docker --version # >= 24.0
docker compose version # >= 2.20
# Node.js & pnpm
node --version # >= 20.0
pnpm --version # >= 10.0
# Optional: Kubernetes tools
kubectl version --client # >= 1.28
helm version # >= 3.12
Hardware Requirements#
| Environment | CPU | RAM | Storage |
|---|---|---|---|
| Development | 4 cores | 8 GB | 50 GB SSD |
| Staging | 8 cores | 16 GB | 200 GB SSD |
| Production | 16+ cores | 32+ GB | 500+ GB SSD |
Network Requirements#
| Port | Service | Access |
|---|---|---|
| 80/443 | HTTP/HTTPS | Public |
| 3000 | API Server | Internal/Load Balancer |
| 5432 | PostgreSQL | Internal only |
| 6379 | Redis | Internal only |
| 9000/9001 | MinIO | Internal only |
Local Development#
Quick Start#
bash
# 1. Clone repository
git clone https://github.com/yemaya/yemaya.git
cd yemaya
# 2. Copy environment file
cp .env.example .env
# 3. Start infrastructure
docker compose up -d
# 4. Install dependencies
pnpm install
# 5. Run database migrations
pnpm --filter @yemaya/database db:migrate
# 6. Start development servers
pnpm dev
Environment Configuration#
Edit .env with your local settings:
bash
# Database
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya
# Redis
REDIS_URL=redis://localhost:6379
# MinIO
MINIO_ENDPOINT=localhost
MINIO_PORT=9000
MINIO_ACCESS_KEY=yemaya_minio
MINIO_SECRET_KEY=yemaya_minio_secret
# Authentication
JWT_SECRET=development-jwt-secret-change-in-production-32chars
# AI Providers (optional)
ANTHROPIC_API_KEY=sk-ant-your-key-here
OPENAI_API_KEY=sk-your-key-here
Docker Compose Profiles#
bash
# Core services only
docker compose up -d
# With Vault (secret management)
docker compose --profile vault up -d
# With ELK Stack (logging)
docker compose --profile logging up -d
# With Prometheus/Grafana (monitoring)
docker compose --profile monitoring up -d
# With backup service
docker compose --profile backup up -d
# All services
docker compose --profile vault --profile logging --profile monitoring up -d
Available Services#
| Service | URL | Credentials |
|---|---|---|
| API Server | http://localhost:3000 | - |
| API Docs (Swagger) | http://localhost:3000/docs | - |
| API Reference (Scalar) | http://localhost:3000/reference | - |
| MinIO Console | http://localhost:9001 | yemaya_minio / yemaya_minio_secret |
| Mailpit (Email) | http://localhost:8025 | - |
| Vault UI | http://localhost:8200 | Token: yemaya-dev-token |
| Kibana | http://localhost:5601 | elastic / yemaya_elastic_dev |
| Grafana | http://localhost:3001 | admin / yemaya_grafana_dev |
| Prometheus | http://localhost:9090 | - |
Development Commands#
bash
# Run specific app
pnpm --filter @yemaya/api dev
pnpm --filter @yemaya/web dev
pnpm --filter @yemaya/workers dev
# Run tests
pnpm test
pnpm --filter @yemaya/core test:watch
# Database operations
pnpm --filter @yemaya/database db:migrate
pnpm --filter @yemaya/database db:studio
pnpm --filter @yemaya/database db:generate
# Build all packages
pnpm build
# Type checking
pnpm typecheck
# Linting
pnpm lint
pnpm lint:fix
Staging Deployment#
Infrastructure Setup#
-
Provision servers (2 app servers, 1 database server, 1 storage server)
-
Configure environment variables:
bash
# /etc/yemaya/.env.staging
NODE_ENV=staging
# Database
POSTGRES_USER=yemaya_staging
POSTGRES_PASSWORD=<generated-password>
POSTGRES_DB=yemaya_staging
DATABASE_URL=postgresql://yemaya_staging:<password>@db-staging:5432/yemaya_staging
# Redis
REDIS_PASSWORD=<generated-password>
REDIS_URL=redis://:<password>@redis-staging:6379
# MinIO
MINIO_ACCESS_KEY=<generated-key>
MINIO_SECRET_KEY=<generated-secret>
# JWT
JWT_SECRET=<generated-secret-32-chars>
# AI Keys
ANTHROPIC_API_KEY=sk-ant-<staging-key>
OPENAI_API_KEY=sk-<staging-key>
- Deploy with Docker Compose:
bash
# On staging server
docker compose -f docker-compose.prod.yml --env-file .env.staging up -d
Continuous Deployment#
yaml
# .github/workflows/deploy-staging.yml
name: Deploy to Staging
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build Docker images
run: |
docker compose -f docker-compose.prod.yml build
- name: Push to registry
run: |
docker tag yemaya-api:latest ${{ secrets.REGISTRY }}/yemaya-api:staging
docker push ${{ secrets.REGISTRY }}/yemaya-api:staging
- name: Deploy to staging
uses: appleboy/ssh-action@v1.0.0
with:
host: ${{ secrets.STAGING_HOST }}
username: deploy
key: ${{ secrets.DEPLOY_KEY }}
script: |
cd /opt/yemaya
docker compose pull
docker compose -f docker-compose.prod.yml up -d
docker compose exec api pnpm db:migrate
Production Deployment#
Pre-Deployment Checklist#
- All environment variables set
- SSL certificates obtained (Let's Encrypt or custom)
- Database backups configured
- Monitoring and alerting configured
- Rate limiting configured
- Security headers configured
- CDN configured for static assets
Infrastructure Requirements#
yaml
# Minimum production infrastructure
API Servers (2+):
- 4 vCPU, 8 GB RAM
- Auto-scaling group
Worker Servers (3+):
- 8 vCPU, 16 GB RAM (for media processing)
- 4 vCPU, 8 GB RAM (for general workers)
Database:
- PostgreSQL 16
- 8 vCPU, 32 GB RAM
- 500 GB SSD (gp3)
- Multi-AZ deployment
Cache:
- Redis 7
- 2 vCPU, 8 GB RAM
- Cluster mode enabled
Storage:
- MinIO or AWS S3
- 1 TB initial
- Versioning enabled
- Lifecycle policies
Environment Variables#
bash
# /etc/yemaya/.env.production
NODE_ENV=production
# Database (use connection pooler)
DATABASE_URL=postgresql://yemaya:<password>@pgbouncer:6432/yemaya?pgbouncer=true
DATABASE_URL_DIRECT=postgresql://yemaya:<password>@db-primary:5432/yemaya
# Redis (cluster mode)
REDIS_URL=redis://:<password>@redis-cluster:6379
# MinIO (distributed mode) or S3
MINIO_ENDPOINT=storage.yemaya.io
MINIO_PORT=443
MINIO_USE_SSL=true
MINIO_ACCESS_KEY=<production-key>
MINIO_SECRET_KEY=<production-secret>
# Authentication
JWT_SECRET=<production-secret-64-chars>
JWT_EXPIRES_IN=15m
JWT_REFRESH_EXPIRES_IN=7d
# AI Providers
ANTHROPIC_API_KEY=sk-ant-<production-key>
OPENAI_API_KEY=sk-<production-key>
# Email (production SMTP)
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASS=<sendgrid-api-key>
SMTP_FROM=noreply@yemaya.io
# Monitoring
SENTRY_DSN=https://<key>@sentry.io/<project>
Deployment Steps#
bash
# 1. Build production images
docker compose -f docker-compose.prod.yml build
# 2. Push to container registry
docker tag yemaya-api:latest registry.yemaya.io/yemaya-api:v1.0.0
docker push registry.yemaya.io/yemaya-api:v1.0.0
# 3. Deploy database migrations (from bastion host)
docker compose exec api pnpm db:migrate
# 4. Rolling update (zero-downtime)
docker compose -f docker-compose.prod.yml up -d --no-deps --scale api=2
# 5. Health check
curl https://api.yemaya.io/health
Nginx Configuration#
nginx
# /etc/nginx/sites-available/yemaya.conf
upstream yemaya_api {
least_conn;
server api-1:3000 weight=1 max_fails=3 fail_timeout=30s;
server api-2:3000 weight=1 max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen 80;
server_name api.yemaya.io;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name api.yemaya.io;
ssl_certificate /etc/letsencrypt/live/api.yemaya.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/api.yemaya.io/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# Security headers
add_header X-Frame-Options DENY always;
add_header X-Content-Type-Options nosniff always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header Content-Security-Policy "default-src 'self'" always;
# Gzip compression
gzip on;
gzip_types text/plain application/json application/javascript text/css;
# Rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;
limit_req zone=api burst=200 nodelay;
location / {
proxy_pass http://yemaya_api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
# Timeouts
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
location /health {
proxy_pass http://yemaya_api;
access_log off;
}
# WebSocket endpoint
location /ws {
proxy_pass http://yemaya_api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
}
Kubernetes Deployment#
Helm Chart Structure#
text
charts/yemaya/
├── Chart.yaml
├── values.yaml
├── values-staging.yaml
├── values-production.yaml
├── templates/
│ ├── _helpers.tpl
│ ├── configmap.yaml
│ ├── secret.yaml
│ ├── deployment-api.yaml
│ ├── deployment-workers.yaml
│ ├── service.yaml
│ ├── ingress.yaml
│ ├── hpa.yaml
│ └── pdb.yaml
values.yaml#
yaml
# charts/yemaya/values.yaml
replicaCount:
api: 2
workers: 3
image:
repository: registry.yemaya.io/yemaya
pullPolicy: IfNotPresent
tag: 'latest'
api:
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 500m
memory: 1Gi
workers:
resources:
limits:
cpu: 4000m
memory: 8Gi
requests:
cpu: 1000m
memory: 2Gi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rate-limit: '100'
hosts:
- host: api.yemaya.io
paths:
- path: /
pathType: Prefix
tls:
- secretName: yemaya-tls
hosts:
- api.yemaya.io
postgresql:
enabled: true
auth:
postgresPassword: ''
username: yemaya
password: ''
database: yemaya
primary:
persistence:
size: 100Gi
readReplicas:
replicaCount: 1
redis:
enabled: true
auth:
password: ''
master:
persistence:
size: 10Gi
minio:
enabled: true
auth:
rootUser: ''
rootPassword: ''
persistence:
size: 500Gi
Deployment Commands#
bash
# Add Helm repositories
helm repo add bitnami https://charts.bitnami.com/bitnami
# Create namespace
kubectl create namespace yemaya
# Create secrets
kubectl create secret generic yemaya-secrets \
--from-literal=jwt-secret=<secret> \
--from-literal=anthropic-api-key=<key> \
-n yemaya
# Install with Helm
helm install yemaya ./charts/yemaya \
-f ./charts/yemaya/values-production.yaml \
-n yemaya
# Upgrade deployment
helm upgrade yemaya ./charts/yemaya \
-f ./charts/yemaya/values-production.yaml \
-n yemaya
# Check status
kubectl get pods -n yemaya
kubectl get ingress -n yemaya
Horizontal Pod Autoscaler#
yaml
# templates/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "yemaya.fullname" . }}-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "yemaya.fullname" . }}-api
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 10
periodSeconds: 60
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
- type: Pods
value: 4
periodSeconds: 15
Monitoring & Observability#
ELK Stack Setup#
The development docker-compose includes full ELK stack:
bash
# Start with logging profile
docker compose --profile logging up -d
# Access Kibana
open http://localhost:5601
# Login: elastic / yemaya_elastic_dev
Log Configuration#
Application logs are structured JSON:
json
{
"@timestamp": "2024-01-15T10:30:00.000Z",
"level": "info",
"service": "api",
"environment": "production",
"message": "Request completed",
"correlation_id": "req-abc123",
"user_id": "usr-xyz789",
"http": {
"method": "GET",
"path": "/v1/projects",
"status_code": 200
},
"duration_ms": 45
}
Prometheus Metrics#
Exposed at /metrics:
text
# API metrics
http_requests_total{method="GET", path="/v1/projects", status="200"}
http_request_duration_seconds_bucket{le="0.1"}
http_active_connections
# Worker metrics
worker_jobs_processed_total{queue="agent", status="completed"}
worker_jobs_duration_seconds_bucket{le="10"}
worker_active_jobs{queue="media"}
# Database metrics
db_connections_active
db_queries_total
db_query_duration_seconds
# Cache metrics
redis_operations_total{operation="get"}
redis_hit_ratio
Grafana Dashboards#
Pre-configured dashboards available:
- API Overview - Request rates, latencies, error rates
- Worker Health - Job processing, queue depths
- Database Performance - Connections, query times
- Infrastructure - CPU, memory, disk usage
Alerting Rules#
yaml
# infrastructure/prometheus/rules/alerts.yml
groups:
- name: yemaya-alerts
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.1
for: 5m
labels:
severity: critical
annotations:
summary: High error rate detected
description: Error rate is {{ $value }}
- alert: SlowResponses
expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 2
for: 5m
labels:
severity: warning
annotations:
summary: Slow response times
description: 95th percentile latency is {{ $value }}s
- alert: WorkerQueueBacklog
expr: worker_queue_depth > 1000
for: 10m
labels:
severity: warning
annotations:
summary: Worker queue backlog growing
description: Queue {{ $labels.queue }} has {{ $value }} pending jobs
- alert: DatabaseConnectionsHigh
expr: db_connections_active / db_connections_max > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: Database connections running high
description: {{ $value | humanizePercentage }} of max connections used
Scaling Strategies#
Horizontal Scaling#
yaml
# API scaling based on CPU/memory
api:
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 20
targetCPU: 70
targetMemory: 80
# Worker scaling based on queue depth
workers:
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 50
metrics:
- type: External
external:
metric:
name: worker_queue_depth
target:
type: Value
value: 100
Database Scaling#
- Read Replicas: Route read queries to replicas
- Connection Pooling: Use PgBouncer for connection management
- Partitioning: Partition large tables by date
yaml
# PgBouncer configuration
pgbouncer:
pool_mode: transaction
max_client_conn: 1000
default_pool_size: 20
reserve_pool_size: 5
reserve_pool_timeout: 3
Cache Scaling#
yaml
# Redis Cluster configuration
redis:
cluster:
enabled: true
nodes: 6
replicas: 1
master:
resources:
limits:
memory: 4Gi
Backup & Recovery#
Database Backups#
bash
# Automated daily backups
docker compose run --rm backup pg_dump -Fc > /backups/yemaya-$(date +%Y%m%d).dump
# Restore from backup
docker compose exec postgres pg_restore -d yemaya /backups/yemaya-20240115.dump
Backup Configuration#
yaml
# infrastructure/backup/config.yml
schedule:
full: '0 2 * * *' # Daily at 2 AM
incremental: '0 * * * *' # Hourly
retention:
days: 30
weekly: 12
monthly: 12
storage:
type: s3
bucket: yemaya-backups
encryption: AES256
notifications:
slack:
webhook: ${SLACK_WEBHOOK}
on_failure: true
on_success: false
Disaster Recovery#
- RTO (Recovery Time Objective): 1 hour
- RPO (Recovery Point Objective): 1 hour
bash
# Disaster recovery procedure
1. Restore database from latest backup
2. Deploy application to new infrastructure
3. Update DNS to point to new endpoints
4. Verify all services healthy
5. Notify stakeholders
Security Hardening#
Secret Management#
Use HashiCorp Vault for production secrets:
bash
# Enable Vault
docker compose --profile vault up -d
# Store secrets
vault kv put secret/yemaya/production \
jwt_secret=<secret> \
anthropic_api_key=<key> \
database_password=<password>
# Application reads from Vault
VAULT_ADDR=https://vault.yemaya.io:8200
VAULT_TOKEN=<app-token>
Network Security#
yaml
# Network policies (Kubernetes)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: yemaya-network-policy
spec:
podSelector:
matchLabels:
app: yemaya-api
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- protocol: TCP
port: 3000
egress:
- to:
- podSelector:
matchLabels:
app: postgresql
ports:
- protocol: TCP
port: 5432
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
Security Checklist#
- All secrets stored in Vault or K8s secrets
- TLS 1.3 enabled on all endpoints
- Rate limiting configured
- CORS properly configured
- SQL injection prevention (parameterized queries)
- XSS prevention (Content-Security-Policy)
- CSRF tokens for mutations
- Audit logging enabled
- Container images scanned
- Dependency vulnerabilities checked
Troubleshooting#
See Troubleshooting Guide for common issues and solutions.
Quick Reference#
Common Commands#
bash
# View logs
docker compose logs -f api
docker compose logs -f workers
# Restart service
docker compose restart api
# Scale workers
docker compose up -d --scale workers=5
# Database shell
docker compose exec postgres psql -U yemaya -d yemaya
# Redis CLI
docker compose exec redis redis-cli
# Run migrations
docker compose exec api pnpm db:migrate
# Health check
curl http://localhost:3000/health
curl http://localhost:3000/health/ready
Environment URLs#
| Environment | API URL | Docs URL |
|---|---|---|
| Development | http://localhost:3000 | http://localhost:3000/docs |
| Staging | https://api.staging.yemaya.io | https://api.staging.yemaya.io/docs |
| Production | https://api.yemaya.io | https://api.yemaya.io/docs |