Eve library

Troubleshooting Guide

Guide

Revision 1

On this page

Common issues and solutions for the Yemaya platform.

Table of Contents#

  1. Quick Diagnostics
  2. Installation Issues
  3. Database Issues
  4. Redis/Cache Issues
  5. Storage Issues
  6. API Server Issues
  7. Worker Issues
  8. Authentication Issues
  9. 3D Generation Issues
  10. WebSocket Issues
  11. Performance Issues
  12. Docker Issues
  13. Kubernetes Issues
  14. Logging & Debugging

Quick Diagnostics#

System Health Check#

bash
# Check all services
docker compose ps

# Check API health
curl http://localhost:3000/health
curl http://localhost:3000/health/ready

# Check database connection
docker compose exec postgres pg_isready -U yemaya

# Check Redis
docker compose exec redis redis-cli ping

# Check MinIO
curl http://localhost:9000/minio/health/live

# View logs
docker compose logs --tail=100 api
docker compose logs --tail=100 workers

Common Quick Fixes#

bash
# Restart all services
docker compose restart

# Rebuild containers
docker compose build --no-cache
docker compose up -d

# Clear all data and start fresh
docker compose down -v
docker compose up -d

# Force recreate containers
docker compose up -d --force-recreate

Installation Issues#

Problem: pnpm install fails with peer dependency errors#

Symptoms:

text
ERR_PNPM_PEER_DEP_ISSUES  Unmet peer dependencies

Solution:

bash
# Clear cache and reinstall
rm -rf node_modules
pnpm store prune
pnpm install --force

Problem: Docker Compose version incompatible#

Symptoms:

text
ERROR: Version in "./docker-compose.yml" is unsupported

Solution:

bash
# Update Docker Compose
sudo apt-get update
sudo apt-get install docker-compose-plugin

# Verify version
docker compose version  # Should be >= 2.20

Problem: Port already in use#

Symptoms:

text
Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use

Solution:

bash
# Find process using port
lsof -i :5432

# Kill process
kill -9 <PID>

# Or change port in docker-compose.yml
ports:
  - "5433:5432"  # Use 5433 instead

Problem: Permission denied errors on Linux#

Symptoms:

text
permission denied while trying to connect to the Docker daemon socket

Solution:

bash
# Add user to docker group
sudo usermod -aG docker $USER

# Apply changes (logout/login or run)
newgrp docker

Database Issues#

Problem: Database connection refused#

Symptoms:

text
Error: connect ECONNREFUSED 127.0.0.1:5432

Causes & Solutions:

  1. PostgreSQL not running:
bash
docker compose up -d postgres
docker compose logs postgres
  1. Wrong connection string:
bash
# Check DATABASE_URL in .env
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya

# For Docker internal connections, use service name
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@postgres:5432/yemaya
  1. PgBouncer issues:
bash
# Connect directly to PostgreSQL, bypassing PgBouncer
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya

Problem: Too many connections#

Symptoms:

text
FATAL: too many connections for role "yemaya"

Solution:

bash
# Check current connections
docker compose exec postgres psql -U yemaya -c "SELECT count(*) FROM pg_stat_activity;"

# Increase max connections in PostgreSQL
# docker-compose.yml
command: postgres -c max_connections=200

# Or use PgBouncer for connection pooling (recommended)
docker compose up -d pgbouncer

Problem: Migration fails#

Symptoms:

text
Error: P3009 migrate found failed migrations

Solution:

bash
# Reset migrations (DEVELOPMENT ONLY - data loss!)
pnpm --filter @yemaya/database db:reset

# Or manually fix
docker compose exec postgres psql -U yemaya -d yemaya
> DELETE FROM "_prisma_migrations" WHERE finished_at IS NULL;
> \q

pnpm --filter @yemaya/database db:migrate

Problem: Database slow queries#

Symptoms:

  • API responses taking > 1s
  • Timeout errors

Diagnosis:

bash
# Check slow queries
docker compose exec postgres psql -U yemaya -d yemaya << EOF
SELECT
  pid,
  now() - pg_stat_activity.query_start AS duration,
  query
FROM pg_stat_activity
WHERE state != 'idle'
  AND now() - pg_stat_activity.query_start > interval '1 second'
ORDER BY duration DESC;
EOF

Solution:

bash
# Add missing indexes
docker compose exec postgres psql -U yemaya -d yemaya << EOF
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_project_id ON assets(project_id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_activity_created_at ON activity_logs(created_at);
ANALYZE;
EOF

Redis/Cache Issues#

Problem: Redis connection refused#

Symptoms:

text
Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED

Solution:

bash
# Check Redis is running
docker compose up -d redis

# Verify connection
docker compose exec redis redis-cli ping
# Should return: PONG

Problem: Redis out of memory#

Symptoms:

text
OOM command not allowed when used memory > 'maxmemory'

Solution:

bash
# Check memory usage
docker compose exec redis redis-cli INFO memory

# Clear cache (safe)
docker compose exec redis redis-cli FLUSHDB

# Increase maxmemory in docker-compose.yml
command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru

Problem: BullMQ jobs stuck#

Symptoms:

  • Jobs not processing
  • Queue depth keeps growing

Diagnosis:

bash
# Check queue status
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:wait"
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:active"

Solution:

bash
# Restart workers
docker compose restart workers

# Clear stuck jobs (CAUTION: data loss)
docker compose exec redis redis-cli KEYS "bull:yemaya:*" | xargs docker compose exec redis redis-cli DEL

Storage Issues#

Problem: MinIO connection failed#

Symptoms:

text
Error: connect ECONNREFUSED 127.0.0.1:9000

Solution:

bash
# Start MinIO
docker compose up -d minio

# Check health
curl http://localhost:9000/minio/health/live

# Verify buckets exist
docker compose exec minio mc alias set yemaya http://localhost:9000 yemaya_minio yemaya_minio_secret
docker compose exec minio mc ls yemaya

Problem: Upload fails with "bucket not found"#

Symptoms:

text
NoSuchBucket: The specified bucket does not exist

Solution:

bash
# Create buckets
docker compose exec minio mc mb yemaya/assets --ignore-existing
docker compose exec minio mc mb yemaya/uploads --ignore-existing
docker compose exec minio mc mb yemaya/renders --ignore-existing

# Or run init container
docker compose up minio-init

Problem: Files not accessible publicly#

Symptoms:

  • 403 Forbidden when accessing asset URLs

Solution:

bash
# Set public read policy on assets bucket
docker compose exec minio mc anonymous set download yemaya/assets

Problem: Storage full#

Symptoms:

text
disk quota exceeded

Solution:

bash
# Check usage
docker compose exec minio mc du yemaya/

# Clean up old uploads
docker compose exec minio mc rm --recursive --older-than 7d yemaya/uploads/

# Or expand volume
docker volume rm yemaya_minio_data
# Recreate with larger size

API Server Issues#

Problem: 503 Service Unavailable#

Symptoms:

  • API returns 503 status
  • Nginx shows upstream errors

Diagnosis:

bash
# Check API container
docker compose ps api
docker compose logs --tail=50 api

# Check if API is responding
docker compose exec api curl localhost:3000/health

Solution:

bash
# Restart API
docker compose restart api

# If memory issue
# Increase memory limit in docker-compose.yml
deploy:
  resources:
    limits:
      memory: 4G

Problem: 429 Too Many Requests#

Symptoms:

json
{
  "error": "rate_limit_exceeded",
  "message": "Too many requests"
}

Solution:

bash
# Wait for rate limit to reset (check X-RateLimit-Reset header)

# Or adjust rate limits in API configuration
# apps/api/src/middleware/rate-limit.ts

Problem: Request timeout#

Symptoms:

text
Error: ETIMEDOUT

Diagnosis:

bash
# Check API responsiveness
time curl http://localhost:3000/health

# Check for blocking queries
docker compose exec postgres psql -U yemaya -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"

Solution:

bash
# Increase timeout in nginx
proxy_read_timeout 120s;

# Optimize slow endpoint (add indexes, caching, pagination)

Worker Issues#

Problem: Workers not processing jobs#

Symptoms:

  • Jobs remain in "waiting" state
  • No worker logs

Diagnosis:

bash
# Check worker status
docker compose ps workers
docker compose logs --tail=50 workers

# Check Redis connection
docker compose exec workers ping redis

Solution:

bash
# Restart workers
docker compose restart workers

# Check environment variables
docker compose exec workers env | grep REDIS

Problem: Job processing failures#

Symptoms:

text
Job failed with error: ...

Diagnosis:

bash
# Check worker logs
docker compose logs workers | grep -i error

# Check specific job
docker compose exec redis redis-cli HGET "bull:yemaya:agent:<job-id>" failedReason

Solution:

  • Fix the underlying error (API key, permission, resource)
  • Retry the job via API or admin interface

Problem: Worker memory leak#

Symptoms:

  • Worker memory grows continuously
  • Eventually OOM killed

Solution:

bash
# Restart workers with memory limit
docker compose up -d --force-recreate workers

# Add memory limit
deploy:
  resources:
    limits:
      memory: 4G

# Enable garbage collection logging
NODE_OPTIONS="--expose-gc --max-old-space-size=4096"

Authentication Issues#

Problem: Invalid JWT token#

Symptoms:

json
{
  "error": "INVALID_TOKEN",
  "message": "Token is invalid or expired"
}

Causes & Solutions:

  1. Token expired:
bash
# Use refresh token to get new access token
curl -X POST http://localhost:3000/v1/auth/refresh \
  -H "Content-Type: application/json" \
  -d '{"refreshToken": "<refresh-token>"}'
  1. Wrong JWT secret:
bash
# Ensure JWT_SECRET matches in all services
grep JWT_SECRET .env
  1. Token from different environment:
bash
# Tokens are environment-specific - use correct environment

Problem: OAuth callback fails#

Symptoms:

  • Redirect loop
  • "Invalid state" error

Solution:

bash
# Verify OAuth redirect URI matches exactly
# In .env
OAUTH_GOOGLE_REDIRECT_URI=http://localhost:3000/v1/auth/oauth/google/callback

# In Google Cloud Console, add exact same URI

Problem: Password reset email not received#

Symptoms:

  • No email in inbox or spam

Diagnosis:

bash
# Check Mailpit (development)
open http://localhost:8025

# Check SMTP configuration
docker compose exec api env | grep SMTP

Solution:

bash
# Ensure SMTP is configured
SMTP_HOST=mailpit
SMTP_PORT=1025

# For production, use real SMTP service
SMTP_HOST=smtp.sendgrid.net

3D Generation Issues#

Problem: Generation job stuck in "processing"#

Symptoms:

  • Job status never changes from "processing"

Diagnosis:

bash
# Check 3D worker logs
docker compose logs workers | grep "3d-generation"

# Check external provider status
curl https://api.meshy.ai/health  # Example

Solution:

bash
# Check API key is valid
grep -E "RODIN|MESHY|TRIPO" .env

# Restart workers
docker compose restart workers

# Cancel stuck job via API
curl -X POST http://localhost:3000/v1/3d/jobs/<job-id>/cancel

Problem: Generated model is low quality#

Symptoms:

  • Mesh has holes or artifacts
  • Textures are blurry

Solution:

bash
# Request higher quality level
{
  "prompt": "...",
  "quality": "high",  # or "ultra"
  "meshProcessing": {
    "optimize": true,
    "targetPolygons": 100000
  }
}

Problem: Provider API rate limited#

Symptoms:

json
{
  "error": "PROVIDER_RATE_LIMITED",
  "message": "Too many requests to 3D provider"
}

Solution:

bash
# Use different provider
{
  "provider": "tripo"  # Instead of "meshy"
}

# Or wait and retry with exponential backoff

WebSocket Issues#

Problem: WebSocket connection fails#

Symptoms:

text
WebSocket connection to 'wss://...' failed

Diagnosis:

bash
# Test WebSocket locally
websocat ws://localhost:3000/ws

# Check nginx WebSocket config
grep -A5 "location /ws" /etc/nginx/nginx.conf

Solution:

nginx
# Ensure nginx has WebSocket support
location /ws {
    proxy_pass http://api;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 86400;
}

Problem: WebSocket disconnects frequently#

Symptoms:

  • Connection drops every few minutes

Solution:

bash
# Increase timeout in nginx
proxy_read_timeout 86400;

# Enable keep-alive
proxy_set_header Connection "";

# Client should implement reconnection logic

Problem: Real-time updates not received#

Symptoms:

  • WebSocket connected but no events

Diagnosis:

bash
# Check if events are being published
docker compose logs api | grep "websocket"
docker compose logs api | grep "emit"

Solution:

bash
# Verify subscription
# Client must send: { "type": "subscribe", "channel": "project:abc123" }

# Check Redis pub/sub
docker compose exec redis redis-cli PUBSUB CHANNELS

Performance Issues#

Problem: Slow API responses#

Diagnosis:

bash
# Enable debug logging
LOG_LEVEL=debug docker compose up -d api

# Check slow queries
docker compose exec postgres psql -U yemaya << EOF
SELECT
  mean_exec_time,
  calls,
  query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
EOF

Solutions:

  1. Add database indexes:
sql
CREATE INDEX CONCURRENTLY idx_xxx ON table(column);
  1. Enable caching:
typescript
// Use Redis caching for expensive queries
const cached = await redis.get(`cache:${key}`);
if (cached) return JSON.parse(cached);
  1. Optimize queries:
typescript
// Use pagination
{ page: 1, limit: 20 }

// Select only needed fields
select: { id: true, name: true }

// Use proper joins
include: { relation: true }

Problem: High memory usage#

Diagnosis:

bash
# Check container memory
docker stats

# Check Node.js heap
docker compose exec api node -e "console.log(process.memoryUsage())"

Solution:

bash
# Increase memory limits
deploy:
  resources:
    limits:
      memory: 4G

# Enable garbage collection
NODE_OPTIONS="--max-old-space-size=4096"

# Check for memory leaks
# Use --inspect and Chrome DevTools
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js

Problem: High CPU usage#

Diagnosis:

bash
# Check CPU usage
docker stats --format "table {{.Name}}\t{{.CPUPerc}}"

# Profile Node.js
docker compose exec api node --prof dist/index.js

Solution:

bash
# Scale horizontally
docker compose up -d --scale api=4

# Optimize CPU-intensive operations
# Move to worker queue

Docker Issues#

Problem: Container keeps restarting#

Diagnosis:

bash
# Check exit code
docker compose ps
docker inspect yemaya-api --format='{{.State.ExitCode}}'

# Check logs
docker compose logs --tail=100 api

Common exit codes:

  • 0: Normal exit
  • 1: Application error
  • 137: OOM killed (increase memory)
  • 143: SIGTERM (normal shutdown)

Problem: Volumes not persisting#

Symptoms:

  • Data lost after restart

Solution:

bash
# Check volume exists
docker volume ls | grep yemaya

# Inspect volume
docker volume inspect yemaya_postgres_data

# Ensure volume is mounted correctly
volumes:
  - postgres_data:/var/lib/postgresql/data  # Named volume
  # NOT - ./data:/var/lib/postgresql/data   # Bind mount (check permissions)

Problem: Network issues between containers#

Diagnosis:

bash
# Check network
docker network ls
docker network inspect yemaya_backend

# Test connectivity
docker compose exec api ping postgres
docker compose exec api nc -zv postgres 5432

Solution:

bash
# Ensure services are on same network
networks:
  - backend

# Recreate networks
docker compose down
docker network prune
docker compose up -d

Kubernetes Issues#

Problem: Pod CrashLoopBackOff#

Diagnosis:

bash
# Check pod status
kubectl describe pod yemaya-api-xxx -n yemaya

# Check logs
kubectl logs yemaya-api-xxx -n yemaya --previous

Common causes:

  • Missing secrets/configmaps
  • Health check failing
  • Application crash

Solution:

bash
# Fix the underlying issue based on logs

# Verify secrets exist
kubectl get secrets -n yemaya

# Check configmap
kubectl get configmap -n yemaya

Problem: Service not reachable#

Diagnosis:

bash
# Check service
kubectl get svc -n yemaya
kubectl describe svc yemaya-api -n yemaya

# Check endpoints
kubectl get endpoints yemaya-api -n yemaya

# Test from within cluster
kubectl run debug --image=curlimages/curl -it --rm -- curl yemaya-api:3000/health

Problem: Ingress not working#

Diagnosis:

bash
# Check ingress
kubectl get ingress -n yemaya
kubectl describe ingress yemaya -n yemaya

# Check ingress controller logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller

Solution:

bash
# Verify TLS secret exists
kubectl get secret yemaya-tls -n yemaya

# Check ingress class
kubectl get ingressclass

Logging & Debugging#

Enable Debug Logging#

bash
# Set log level to debug
LOG_LEVEL=debug docker compose up -d api

# Enable pretty printing
LOG_PRETTY=true docker compose up -d api

View Structured Logs#

bash
# Parse JSON logs
docker compose logs api | jq 'select(.level == "error")'

# Filter by correlation ID
docker compose logs api | jq 'select(.correlation_id == "req-abc123")'

Remote Debugging#

bash
# Enable Node.js inspector
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js

# Connect Chrome DevTools
chrome://inspect

# Or VS Code
{
  "type": "node",
  "request": "attach",
  "port": 9229,
  "remoteRoot": "/app"
}

Database Query Logging#

bash
# Enable query logging
docker compose exec postgres psql -U yemaya << EOF
ALTER SYSTEM SET log_statement = 'all';
ALTER SYSTEM SET log_min_duration_statement = 0;
SELECT pg_reload_conf();
EOF

# View query logs
docker compose logs postgres | grep "LOG:  statement:"

Request Tracing#

bash
# All requests have correlation IDs
# Check X-Request-ID header in response

# Search logs by correlation ID
docker compose logs api | grep "req-abc123"

Getting Help#

If you can't resolve an issue:

  1. Check documentation:

  2. Search existing issues:

    • https://github.com/yemaya/yemaya/issues
  3. Create a new issue with:

    • Environment (OS, Docker version, Node version)
    • Steps to reproduce
    • Expected vs actual behavior
    • Relevant logs (sanitized of secrets)
  4. Community support:

    • Discord: https://discord.gg/yemaya
    • Discussions: https://github.com/yemaya/yemaya/discussions

Quick Reference#

Health Check URLs#

Service URL Expected
API http://localhost:3000/health {"status":"healthy"}
API Ready http://localhost:3000/health/ready {"status":"ready"}
MinIO http://localhost:9000/minio/health/live HTTP 200
Kibana http://localhost:5601/api/status {"level":"available"}

Log Locations#

Container Log Path
API stdout (docker logs)
Workers stdout (docker logs)
PostgreSQL /var/lib/postgresql/data/log/
Nginx /var/log/nginx/
Elasticsearch /var/log/elasticsearch/

Environment Variables Reference#

Variable Required Description
DATABASE_URL Yes PostgreSQL connection string
REDIS_URL Yes Redis connection string
JWT_SECRET Yes JWT signing secret (32+ chars)
MINIO_* Yes MinIO configuration
ANTHROPIC_API_KEY For AI Claude API key
OPENAI_API_KEY For AI OpenAI API key