Common issues and solutions for the Yemaya platform.
Table of Contents#
- Quick Diagnostics
- Installation Issues
- Database Issues
- Redis/Cache Issues
- Storage Issues
- API Server Issues
- Worker Issues
- Authentication Issues
- 3D Generation Issues
- WebSocket Issues
- Performance Issues
- Docker Issues
- Kubernetes Issues
- Logging & Debugging
Quick Diagnostics#
System Health Check#
# Check all services
docker compose ps
# Check API health
curl http://localhost:3000/health
curl http://localhost:3000/health/ready
# Check database connection
docker compose exec postgres pg_isready -U yemaya
# Check Redis
docker compose exec redis redis-cli ping
# Check MinIO
curl http://localhost:9000/minio/health/live
# View logs
docker compose logs --tail=100 api
docker compose logs --tail=100 workers
Common Quick Fixes#
# Restart all services
docker compose restart
# Rebuild containers
docker compose build --no-cache
docker compose up -d
# Clear all data and start fresh
docker compose down -v
docker compose up -d
# Force recreate containers
docker compose up -d --force-recreate
Installation Issues#
Problem: pnpm install fails with peer dependency errors#
Symptoms:
ERR_PNPM_PEER_DEP_ISSUES Unmet peer dependencies
Solution:
# Clear cache and reinstall
rm -rf node_modules
pnpm store prune
pnpm install --force
Problem: Docker Compose version incompatible#
Symptoms:
ERROR: Version in "./docker-compose.yml" is unsupported
Solution:
# Update Docker Compose
sudo apt-get update
sudo apt-get install docker-compose-plugin
# Verify version
docker compose version # Should be >= 2.20
Problem: Port already in use#
Symptoms:
Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use
Solution:
# Find process using port
lsof -i :5432
# Kill process
kill -9 <PID>
# Or change port in docker-compose.yml
ports:
- "5433:5432" # Use 5433 instead
Problem: Permission denied errors on Linux#
Symptoms:
permission denied while trying to connect to the Docker daemon socket
Solution:
# Add user to docker group
sudo usermod -aG docker $USER
# Apply changes (logout/login or run)
newgrp docker
Database Issues#
Problem: Database connection refused#
Symptoms:
Error: connect ECONNREFUSED 127.0.0.1:5432
Causes & Solutions:
- PostgreSQL not running:
docker compose up -d postgres
docker compose logs postgres
- Wrong connection string:
# Check DATABASE_URL in .env
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya
# For Docker internal connections, use service name
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@postgres:5432/yemaya
- PgBouncer issues:
# Connect directly to PostgreSQL, bypassing PgBouncer
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya
Problem: Too many connections#
Symptoms:
FATAL: too many connections for role "yemaya"
Solution:
# Check current connections
docker compose exec postgres psql -U yemaya -c "SELECT count(*) FROM pg_stat_activity;"
# Increase max connections in PostgreSQL
# docker-compose.yml
command: postgres -c max_connections=200
# Or use PgBouncer for connection pooling (recommended)
docker compose up -d pgbouncer
Problem: Migration fails#
Symptoms:
Error: P3009 migrate found failed migrations
Solution:
# Reset migrations (DEVELOPMENT ONLY - data loss!)
pnpm --filter @yemaya/database db:reset
# Or manually fix
docker compose exec postgres psql -U yemaya -d yemaya
> DELETE FROM "_prisma_migrations" WHERE finished_at IS NULL;
> \q
pnpm --filter @yemaya/database db:migrate
Problem: Database slow queries#
Symptoms:
- API responses taking > 1s
- Timeout errors
Diagnosis:
# Check slow queries
docker compose exec postgres psql -U yemaya -d yemaya << EOF
SELECT
pid,
now() - pg_stat_activity.query_start AS duration,
query
FROM pg_stat_activity
WHERE state != 'idle'
AND now() - pg_stat_activity.query_start > interval '1 second'
ORDER BY duration DESC;
EOF
Solution:
# Add missing indexes
docker compose exec postgres psql -U yemaya -d yemaya << EOF
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_project_id ON assets(project_id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_activity_created_at ON activity_logs(created_at);
ANALYZE;
EOF
Redis/Cache Issues#
Problem: Redis connection refused#
Symptoms:
Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED
Solution:
# Check Redis is running
docker compose up -d redis
# Verify connection
docker compose exec redis redis-cli ping
# Should return: PONG
Problem: Redis out of memory#
Symptoms:
OOM command not allowed when used memory > 'maxmemory'
Solution:
# Check memory usage
docker compose exec redis redis-cli INFO memory
# Clear cache (safe)
docker compose exec redis redis-cli FLUSHDB
# Increase maxmemory in docker-compose.yml
command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
Problem: BullMQ jobs stuck#
Symptoms:
- Jobs not processing
- Queue depth keeps growing
Diagnosis:
# Check queue status
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:wait"
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:active"
Solution:
# Restart workers
docker compose restart workers
# Clear stuck jobs (CAUTION: data loss)
docker compose exec redis redis-cli KEYS "bull:yemaya:*" | xargs docker compose exec redis redis-cli DEL
Storage Issues#
Problem: MinIO connection failed#
Symptoms:
Error: connect ECONNREFUSED 127.0.0.1:9000
Solution:
# Start MinIO
docker compose up -d minio
# Check health
curl http://localhost:9000/minio/health/live
# Verify buckets exist
docker compose exec minio mc alias set yemaya http://localhost:9000 yemaya_minio yemaya_minio_secret
docker compose exec minio mc ls yemaya
Problem: Upload fails with "bucket not found"#
Symptoms:
NoSuchBucket: The specified bucket does not exist
Solution:
# Create buckets
docker compose exec minio mc mb yemaya/assets --ignore-existing
docker compose exec minio mc mb yemaya/uploads --ignore-existing
docker compose exec minio mc mb yemaya/renders --ignore-existing
# Or run init container
docker compose up minio-init
Problem: Files not accessible publicly#
Symptoms:
- 403 Forbidden when accessing asset URLs
Solution:
# Set public read policy on assets bucket
docker compose exec minio mc anonymous set download yemaya/assets
Problem: Storage full#
Symptoms:
disk quota exceeded
Solution:
# Check usage
docker compose exec minio mc du yemaya/
# Clean up old uploads
docker compose exec minio mc rm --recursive --older-than 7d yemaya/uploads/
# Or expand volume
docker volume rm yemaya_minio_data
# Recreate with larger size
API Server Issues#
Problem: 503 Service Unavailable#
Symptoms:
- API returns 503 status
- Nginx shows upstream errors
Diagnosis:
# Check API container
docker compose ps api
docker compose logs --tail=50 api
# Check if API is responding
docker compose exec api curl localhost:3000/health
Solution:
# Restart API
docker compose restart api
# If memory issue
# Increase memory limit in docker-compose.yml
deploy:
resources:
limits:
memory: 4G
Problem: 429 Too Many Requests#
Symptoms:
{
"error": "rate_limit_exceeded",
"message": "Too many requests"
}
Solution:
# Wait for rate limit to reset (check X-RateLimit-Reset header)
# Or adjust rate limits in API configuration
# apps/api/src/middleware/rate-limit.ts
Problem: Request timeout#
Symptoms:
Error: ETIMEDOUT
Diagnosis:
# Check API responsiveness
time curl http://localhost:3000/health
# Check for blocking queries
docker compose exec postgres psql -U yemaya -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"
Solution:
# Increase timeout in nginx
proxy_read_timeout 120s;
# Optimize slow endpoint (add indexes, caching, pagination)
Worker Issues#
Problem: Workers not processing jobs#
Symptoms:
- Jobs remain in "waiting" state
- No worker logs
Diagnosis:
# Check worker status
docker compose ps workers
docker compose logs --tail=50 workers
# Check Redis connection
docker compose exec workers ping redis
Solution:
# Restart workers
docker compose restart workers
# Check environment variables
docker compose exec workers env | grep REDIS
Problem: Job processing failures#
Symptoms:
Job failed with error: ...
Diagnosis:
# Check worker logs
docker compose logs workers | grep -i error
# Check specific job
docker compose exec redis redis-cli HGET "bull:yemaya:agent:<job-id>" failedReason
Solution:
- Fix the underlying error (API key, permission, resource)
- Retry the job via API or admin interface
Problem: Worker memory leak#
Symptoms:
- Worker memory grows continuously
- Eventually OOM killed
Solution:
# Restart workers with memory limit
docker compose up -d --force-recreate workers
# Add memory limit
deploy:
resources:
limits:
memory: 4G
# Enable garbage collection logging
NODE_OPTIONS="--expose-gc --max-old-space-size=4096"
Authentication Issues#
Problem: Invalid JWT token#
Symptoms:
{
"error": "INVALID_TOKEN",
"message": "Token is invalid or expired"
}
Causes & Solutions:
- Token expired:
# Use refresh token to get new access token
curl -X POST http://localhost:3000/v1/auth/refresh \
-H "Content-Type: application/json" \
-d '{"refreshToken": "<refresh-token>"}'
- Wrong JWT secret:
# Ensure JWT_SECRET matches in all services
grep JWT_SECRET .env
- Token from different environment:
# Tokens are environment-specific - use correct environment
Problem: OAuth callback fails#
Symptoms:
- Redirect loop
- "Invalid state" error
Solution:
# Verify OAuth redirect URI matches exactly
# In .env
OAUTH_GOOGLE_REDIRECT_URI=http://localhost:3000/v1/auth/oauth/google/callback
# In Google Cloud Console, add exact same URI
Problem: Password reset email not received#
Symptoms:
- No email in inbox or spam
Diagnosis:
# Check Mailpit (development)
open http://localhost:8025
# Check SMTP configuration
docker compose exec api env | grep SMTP
Solution:
# Ensure SMTP is configured
SMTP_HOST=mailpit
SMTP_PORT=1025
# For production, use real SMTP service
SMTP_HOST=smtp.sendgrid.net
3D Generation Issues#
Problem: Generation job stuck in "processing"#
Symptoms:
- Job status never changes from "processing"
Diagnosis:
# Check 3D worker logs
docker compose logs workers | grep "3d-generation"
# Check external provider status
curl https://api.meshy.ai/health # Example
Solution:
# Check API key is valid
grep -E "RODIN|MESHY|TRIPO" .env
# Restart workers
docker compose restart workers
# Cancel stuck job via API
curl -X POST http://localhost:3000/v1/3d/jobs/<job-id>/cancel
Problem: Generated model is low quality#
Symptoms:
- Mesh has holes or artifacts
- Textures are blurry
Solution:
# Request higher quality level
{
"prompt": "...",
"quality": "high", # or "ultra"
"meshProcessing": {
"optimize": true,
"targetPolygons": 100000
}
}
Problem: Provider API rate limited#
Symptoms:
{
"error": "PROVIDER_RATE_LIMITED",
"message": "Too many requests to 3D provider"
}
Solution:
# Use different provider
{
"provider": "tripo" # Instead of "meshy"
}
# Or wait and retry with exponential backoff
WebSocket Issues#
Problem: WebSocket connection fails#
Symptoms:
WebSocket connection to 'wss://...' failed
Diagnosis:
# Test WebSocket locally
websocat ws://localhost:3000/ws
# Check nginx WebSocket config
grep -A5 "location /ws" /etc/nginx/nginx.conf
Solution:
# Ensure nginx has WebSocket support
location /ws {
proxy_pass http://api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
Problem: WebSocket disconnects frequently#
Symptoms:
- Connection drops every few minutes
Solution:
# Increase timeout in nginx
proxy_read_timeout 86400;
# Enable keep-alive
proxy_set_header Connection "";
# Client should implement reconnection logic
Problem: Real-time updates not received#
Symptoms:
- WebSocket connected but no events
Diagnosis:
# Check if events are being published
docker compose logs api | grep "websocket"
docker compose logs api | grep "emit"
Solution:
# Verify subscription
# Client must send: { "type": "subscribe", "channel": "project:abc123" }
# Check Redis pub/sub
docker compose exec redis redis-cli PUBSUB CHANNELS
Performance Issues#
Problem: Slow API responses#
Diagnosis:
# Enable debug logging
LOG_LEVEL=debug docker compose up -d api
# Check slow queries
docker compose exec postgres psql -U yemaya << EOF
SELECT
mean_exec_time,
calls,
query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
EOF
Solutions:
- Add database indexes:
CREATE INDEX CONCURRENTLY idx_xxx ON table(column);
- Enable caching:
// Use Redis caching for expensive queries
const cached = await redis.get(`cache:${key}`);
if (cached) return JSON.parse(cached);
- Optimize queries:
// Use pagination
{ page: 1, limit: 20 }
// Select only needed fields
select: { id: true, name: true }
// Use proper joins
include: { relation: true }
Problem: High memory usage#
Diagnosis:
# Check container memory
docker stats
# Check Node.js heap
docker compose exec api node -e "console.log(process.memoryUsage())"
Solution:
# Increase memory limits
deploy:
resources:
limits:
memory: 4G
# Enable garbage collection
NODE_OPTIONS="--max-old-space-size=4096"
# Check for memory leaks
# Use --inspect and Chrome DevTools
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js
Problem: High CPU usage#
Diagnosis:
# Check CPU usage
docker stats --format "table {{.Name}}\t{{.CPUPerc}}"
# Profile Node.js
docker compose exec api node --prof dist/index.js
Solution:
# Scale horizontally
docker compose up -d --scale api=4
# Optimize CPU-intensive operations
# Move to worker queue
Docker Issues#
Problem: Container keeps restarting#
Diagnosis:
# Check exit code
docker compose ps
docker inspect yemaya-api --format='{{.State.ExitCode}}'
# Check logs
docker compose logs --tail=100 api
Common exit codes:
0: Normal exit1: Application error137: OOM killed (increase memory)143: SIGTERM (normal shutdown)
Problem: Volumes not persisting#
Symptoms:
- Data lost after restart
Solution:
# Check volume exists
docker volume ls | grep yemaya
# Inspect volume
docker volume inspect yemaya_postgres_data
# Ensure volume is mounted correctly
volumes:
- postgres_data:/var/lib/postgresql/data # Named volume
# NOT - ./data:/var/lib/postgresql/data # Bind mount (check permissions)
Problem: Network issues between containers#
Diagnosis:
# Check network
docker network ls
docker network inspect yemaya_backend
# Test connectivity
docker compose exec api ping postgres
docker compose exec api nc -zv postgres 5432
Solution:
# Ensure services are on same network
networks:
- backend
# Recreate networks
docker compose down
docker network prune
docker compose up -d
Kubernetes Issues#
Problem: Pod CrashLoopBackOff#
Diagnosis:
# Check pod status
kubectl describe pod yemaya-api-xxx -n yemaya
# Check logs
kubectl logs yemaya-api-xxx -n yemaya --previous
Common causes:
- Missing secrets/configmaps
- Health check failing
- Application crash
Solution:
# Fix the underlying issue based on logs
# Verify secrets exist
kubectl get secrets -n yemaya
# Check configmap
kubectl get configmap -n yemaya
Problem: Service not reachable#
Diagnosis:
# Check service
kubectl get svc -n yemaya
kubectl describe svc yemaya-api -n yemaya
# Check endpoints
kubectl get endpoints yemaya-api -n yemaya
# Test from within cluster
kubectl run debug --image=curlimages/curl -it --rm -- curl yemaya-api:3000/health
Problem: Ingress not working#
Diagnosis:
# Check ingress
kubectl get ingress -n yemaya
kubectl describe ingress yemaya -n yemaya
# Check ingress controller logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
Solution:
# Verify TLS secret exists
kubectl get secret yemaya-tls -n yemaya
# Check ingress class
kubectl get ingressclass
Logging & Debugging#
Enable Debug Logging#
# Set log level to debug
LOG_LEVEL=debug docker compose up -d api
# Enable pretty printing
LOG_PRETTY=true docker compose up -d api
View Structured Logs#
# Parse JSON logs
docker compose logs api | jq 'select(.level == "error")'
# Filter by correlation ID
docker compose logs api | jq 'select(.correlation_id == "req-abc123")'
Remote Debugging#
# Enable Node.js inspector
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js
# Connect Chrome DevTools
chrome://inspect
# Or VS Code
{
"type": "node",
"request": "attach",
"port": 9229,
"remoteRoot": "/app"
}
Database Query Logging#
# Enable query logging
docker compose exec postgres psql -U yemaya << EOF
ALTER SYSTEM SET log_statement = 'all';
ALTER SYSTEM SET log_min_duration_statement = 0;
SELECT pg_reload_conf();
EOF
# View query logs
docker compose logs postgres | grep "LOG: statement:"
Request Tracing#
# All requests have correlation IDs
# Check X-Request-ID header in response
# Search logs by correlation ID
docker compose logs api | grep "req-abc123"
Getting Help#
If you can't resolve an issue:
-
Check documentation:
-
Search existing issues:
- https://github.com/yemaya/yemaya/issues
-
Create a new issue with:
- Environment (OS, Docker version, Node version)
- Steps to reproduce
- Expected vs actual behavior
- Relevant logs (sanitized of secrets)
-
Community support:
- Discord: https://discord.gg/yemaya
- Discussions: https://github.com/yemaya/yemaya/discussions
Quick Reference#
Health Check URLs#
| Service | URL | Expected |
|---|---|---|
| API | http://localhost:3000/health | {"status":"healthy"} |
| API Ready | http://localhost:3000/health/ready | {"status":"ready"} |
| MinIO | http://localhost:9000/minio/health/live | HTTP 200 |
| Kibana | http://localhost:5601/api/status | {"level":"available"} |
Log Locations#
| Container | Log Path |
|---|---|
| API | stdout (docker logs) |
| Workers | stdout (docker logs) |
| PostgreSQL | /var/lib/postgresql/data/log/ |
| Nginx | /var/log/nginx/ |
| Elasticsearch | /var/log/elasticsearch/ |
Environment Variables Reference#
| Variable | Required | Description |
|---|---|---|
DATABASE_URL |
Yes | PostgreSQL connection string |
REDIS_URL |
Yes | Redis connection string |
JWT_SECRET |
Yes | JWT signing secret (32+ chars) |
MINIO_* |
Yes | MinIO configuration |
ANTHROPIC_API_KEY |
For AI | Claude API key |
OPENAI_API_KEY |
For AI | OpenAI API key |