# Troubleshooting Guide

Common issues and solutions for the Yemaya platform.

## Table of Contents

1. [Quick Diagnostics](#quick-diagnostics)
2. [Installation Issues](#installation-issues)
3. [Database Issues](#database-issues)
4. [Redis/Cache Issues](#rediscache-issues)
5. [Storage Issues](#storage-issues)
6. [API Server Issues](#api-server-issues)
7. [Worker Issues](#worker-issues)
8. [Authentication Issues](#authentication-issues)
9. [3D Generation Issues](#3d-generation-issues)
10. [WebSocket Issues](#websocket-issues)
11. [Performance Issues](#performance-issues)
12. [Docker Issues](#docker-issues)
13. [Kubernetes Issues](#kubernetes-issues)
14. [Logging & Debugging](#logging--debugging)

---

## Quick Diagnostics

### System Health Check

```bash
# Check all services
docker compose ps

# Check API health
curl http://localhost:3000/health
curl http://localhost:3000/health/ready

# Check database connection
docker compose exec postgres pg_isready -U yemaya

# Check Redis
docker compose exec redis redis-cli ping

# Check MinIO
curl http://localhost:9000/minio/health/live

# View logs
docker compose logs --tail=100 api
docker compose logs --tail=100 workers
```

### Common Quick Fixes

```bash
# Restart all services
docker compose restart

# Rebuild containers
docker compose build --no-cache
docker compose up -d

# Clear all data and start fresh
docker compose down -v
docker compose up -d

# Force recreate containers
docker compose up -d --force-recreate
```

---

## Installation Issues

### Problem: `pnpm install` fails with peer dependency errors

**Symptoms:**

```
ERR_PNPM_PEER_DEP_ISSUES  Unmet peer dependencies
```

**Solution:**

```bash
# Clear cache and reinstall
rm -rf node_modules
pnpm store prune
pnpm install --force
```

### Problem: Docker Compose version incompatible

**Symptoms:**

```
ERROR: Version in "./docker-compose.yml" is unsupported
```

**Solution:**

```bash
# Update Docker Compose
sudo apt-get update
sudo apt-get install docker-compose-plugin

# Verify version
docker compose version  # Should be >= 2.20
```

### Problem: Port already in use

**Symptoms:**

```
Error starting userland proxy: listen tcp4 0.0.0.0:5432: bind: address already in use
```

**Solution:**

```bash
# Find process using port
lsof -i :5432

# Kill process
kill -9 <PID>

# Or change port in docker-compose.yml
ports:
  - "5433:5432"  # Use 5433 instead
```

### Problem: Permission denied errors on Linux

**Symptoms:**

```
permission denied while trying to connect to the Docker daemon socket
```

**Solution:**

```bash
# Add user to docker group
sudo usermod -aG docker $USER

# Apply changes (logout/login or run)
newgrp docker
```

---

## Database Issues

### Problem: Database connection refused

**Symptoms:**

```
Error: connect ECONNREFUSED 127.0.0.1:5432
```

**Causes & Solutions:**

1. **PostgreSQL not running:**

```bash
docker compose up -d postgres
docker compose logs postgres
```

2. **Wrong connection string:**

```bash
# Check DATABASE_URL in .env
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya

# For Docker internal connections, use service name
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@postgres:5432/yemaya
```

3. **PgBouncer issues:**

```bash
# Connect directly to PostgreSQL, bypassing PgBouncer
DATABASE_URL=postgresql://yemaya:yemaya_dev_password@localhost:5432/yemaya
```

### Problem: Too many connections

**Symptoms:**

```
FATAL: too many connections for role "yemaya"
```

**Solution:**

```bash
# Check current connections
docker compose exec postgres psql -U yemaya -c "SELECT count(*) FROM pg_stat_activity;"

# Increase max connections in PostgreSQL
# docker-compose.yml
command: postgres -c max_connections=200

# Or use PgBouncer for connection pooling (recommended)
docker compose up -d pgbouncer
```

### Problem: Migration fails

**Symptoms:**

```
Error: P3009 migrate found failed migrations
```

**Solution:**

```bash
# Reset migrations (DEVELOPMENT ONLY - data loss!)
pnpm --filter @yemaya/database db:reset

# Or manually fix
docker compose exec postgres psql -U yemaya -d yemaya
> DELETE FROM "_prisma_migrations" WHERE finished_at IS NULL;
> \q

pnpm --filter @yemaya/database db:migrate
```

### Problem: Database slow queries

**Symptoms:**

- API responses taking > 1s
- Timeout errors

**Diagnosis:**

```bash
# Check slow queries
docker compose exec postgres psql -U yemaya -d yemaya << EOF
SELECT
  pid,
  now() - pg_stat_activity.query_start AS duration,
  query
FROM pg_stat_activity
WHERE state != 'idle'
  AND now() - pg_stat_activity.query_start > interval '1 second'
ORDER BY duration DESC;
EOF
```

**Solution:**

```bash
# Add missing indexes
docker compose exec postgres psql -U yemaya -d yemaya << EOF
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_assets_project_id ON assets(project_id);
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_activity_created_at ON activity_logs(created_at);
ANALYZE;
EOF
```

---

## Redis/Cache Issues

### Problem: Redis connection refused

**Symptoms:**

```
Error: Redis connection to localhost:6379 failed - connect ECONNREFUSED
```

**Solution:**

```bash
# Check Redis is running
docker compose up -d redis

# Verify connection
docker compose exec redis redis-cli ping
# Should return: PONG
```

### Problem: Redis out of memory

**Symptoms:**

```
OOM command not allowed when used memory > 'maxmemory'
```

**Solution:**

```bash
# Check memory usage
docker compose exec redis redis-cli INFO memory

# Clear cache (safe)
docker compose exec redis redis-cli FLUSHDB

# Increase maxmemory in docker-compose.yml
command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
```

### Problem: BullMQ jobs stuck

**Symptoms:**

- Jobs not processing
- Queue depth keeps growing

**Diagnosis:**

```bash
# Check queue status
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:wait"
docker compose exec redis redis-cli LLEN "bull:yemaya:agent:active"
```

**Solution:**

```bash
# Restart workers
docker compose restart workers

# Clear stuck jobs (CAUTION: data loss)
docker compose exec redis redis-cli KEYS "bull:yemaya:*" | xargs docker compose exec redis redis-cli DEL
```

---

## Storage Issues

### Problem: MinIO connection failed

**Symptoms:**

```
Error: connect ECONNREFUSED 127.0.0.1:9000
```

**Solution:**

```bash
# Start MinIO
docker compose up -d minio

# Check health
curl http://localhost:9000/minio/health/live

# Verify buckets exist
docker compose exec minio mc alias set yemaya http://localhost:9000 yemaya_minio yemaya_minio_secret
docker compose exec minio mc ls yemaya
```

### Problem: Upload fails with "bucket not found"

**Symptoms:**

```
NoSuchBucket: The specified bucket does not exist
```

**Solution:**

```bash
# Create buckets
docker compose exec minio mc mb yemaya/assets --ignore-existing
docker compose exec minio mc mb yemaya/uploads --ignore-existing
docker compose exec minio mc mb yemaya/renders --ignore-existing

# Or run init container
docker compose up minio-init
```

### Problem: Files not accessible publicly

**Symptoms:**

- 403 Forbidden when accessing asset URLs

**Solution:**

```bash
# Set public read policy on assets bucket
docker compose exec minio mc anonymous set download yemaya/assets
```

### Problem: Storage full

**Symptoms:**

```
disk quota exceeded
```

**Solution:**

```bash
# Check usage
docker compose exec minio mc du yemaya/

# Clean up old uploads
docker compose exec minio mc rm --recursive --older-than 7d yemaya/uploads/

# Or expand volume
docker volume rm yemaya_minio_data
# Recreate with larger size
```

---

## API Server Issues

### Problem: 503 Service Unavailable

**Symptoms:**

- API returns 503 status
- Nginx shows upstream errors

**Diagnosis:**

```bash
# Check API container
docker compose ps api
docker compose logs --tail=50 api

# Check if API is responding
docker compose exec api curl localhost:3000/health
```

**Solution:**

```bash
# Restart API
docker compose restart api

# If memory issue
# Increase memory limit in docker-compose.yml
deploy:
  resources:
    limits:
      memory: 4G
```

### Problem: 429 Too Many Requests

**Symptoms:**

```json
{
  "error": "rate_limit_exceeded",
  "message": "Too many requests"
}
```

**Solution:**

```bash
# Wait for rate limit to reset (check X-RateLimit-Reset header)

# Or adjust rate limits in API configuration
# apps/api/src/middleware/rate-limit.ts
```

### Problem: Request timeout

**Symptoms:**

```
Error: ETIMEDOUT
```

**Diagnosis:**

```bash
# Check API responsiveness
time curl http://localhost:3000/health

# Check for blocking queries
docker compose exec postgres psql -U yemaya -c "SELECT * FROM pg_stat_activity WHERE state = 'active';"
```

**Solution:**

```bash
# Increase timeout in nginx
proxy_read_timeout 120s;

# Optimize slow endpoint (add indexes, caching, pagination)
```

---

## Worker Issues

### Problem: Workers not processing jobs

**Symptoms:**

- Jobs remain in "waiting" state
- No worker logs

**Diagnosis:**

```bash
# Check worker status
docker compose ps workers
docker compose logs --tail=50 workers

# Check Redis connection
docker compose exec workers ping redis
```

**Solution:**

```bash
# Restart workers
docker compose restart workers

# Check environment variables
docker compose exec workers env | grep REDIS
```

### Problem: Job processing failures

**Symptoms:**

```
Job failed with error: ...
```

**Diagnosis:**

```bash
# Check worker logs
docker compose logs workers | grep -i error

# Check specific job
docker compose exec redis redis-cli HGET "bull:yemaya:agent:<job-id>" failedReason
```

**Solution:**

- Fix the underlying error (API key, permission, resource)
- Retry the job via API or admin interface

### Problem: Worker memory leak

**Symptoms:**

- Worker memory grows continuously
- Eventually OOM killed

**Solution:**

```bash
# Restart workers with memory limit
docker compose up -d --force-recreate workers

# Add memory limit
deploy:
  resources:
    limits:
      memory: 4G

# Enable garbage collection logging
NODE_OPTIONS="--expose-gc --max-old-space-size=4096"
```

---

## Authentication Issues

### Problem: Invalid JWT token

**Symptoms:**

```json
{
  "error": "INVALID_TOKEN",
  "message": "Token is invalid or expired"
}
```

**Causes & Solutions:**

1. **Token expired:**

```bash
# Use refresh token to get new access token
curl -X POST http://localhost:3000/v1/auth/refresh \
  -H "Content-Type: application/json" \
  -d '{"refreshToken": "<refresh-token>"}'
```

2. **Wrong JWT secret:**

```bash
# Ensure JWT_SECRET matches in all services
grep JWT_SECRET .env
```

3. **Token from different environment:**

```bash
# Tokens are environment-specific - use correct environment
```

### Problem: OAuth callback fails

**Symptoms:**

- Redirect loop
- "Invalid state" error

**Solution:**

```bash
# Verify OAuth redirect URI matches exactly
# In .env
OAUTH_GOOGLE_REDIRECT_URI=http://localhost:3000/v1/auth/oauth/google/callback

# In Google Cloud Console, add exact same URI
```

### Problem: Password reset email not received

**Symptoms:**

- No email in inbox or spam

**Diagnosis:**

```bash
# Check Mailpit (development)
open http://localhost:8025

# Check SMTP configuration
docker compose exec api env | grep SMTP
```

**Solution:**

```bash
# Ensure SMTP is configured
SMTP_HOST=mailpit
SMTP_PORT=1025

# For production, use real SMTP service
SMTP_HOST=smtp.sendgrid.net
```

---

## 3D Generation Issues

### Problem: Generation job stuck in "processing"

**Symptoms:**

- Job status never changes from "processing"

**Diagnosis:**

```bash
# Check 3D worker logs
docker compose logs workers | grep "3d-generation"

# Check external provider status
curl https://api.meshy.ai/health  # Example
```

**Solution:**

```bash
# Check API key is valid
grep -E "RODIN|MESHY|TRIPO" .env

# Restart workers
docker compose restart workers

# Cancel stuck job via API
curl -X POST http://localhost:3000/v1/3d/jobs/<job-id>/cancel
```

### Problem: Generated model is low quality

**Symptoms:**

- Mesh has holes or artifacts
- Textures are blurry

**Solution:**

```bash
# Request higher quality level
{
  "prompt": "...",
  "quality": "high",  # or "ultra"
  "meshProcessing": {
    "optimize": true,
    "targetPolygons": 100000
  }
}
```

### Problem: Provider API rate limited

**Symptoms:**

```json
{
  "error": "PROVIDER_RATE_LIMITED",
  "message": "Too many requests to 3D provider"
}
```

**Solution:**

```bash
# Use different provider
{
  "provider": "tripo"  # Instead of "meshy"
}

# Or wait and retry with exponential backoff
```

---

## WebSocket Issues

### Problem: WebSocket connection fails

**Symptoms:**

```
WebSocket connection to 'wss://...' failed
```

**Diagnosis:**

```bash
# Test WebSocket locally
websocat ws://localhost:3000/ws

# Check nginx WebSocket config
grep -A5 "location /ws" /etc/nginx/nginx.conf
```

**Solution:**

```nginx
# Ensure nginx has WebSocket support
location /ws {
    proxy_pass http://api;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_read_timeout 86400;
}
```

### Problem: WebSocket disconnects frequently

**Symptoms:**

- Connection drops every few minutes

**Solution:**

```bash
# Increase timeout in nginx
proxy_read_timeout 86400;

# Enable keep-alive
proxy_set_header Connection "";

# Client should implement reconnection logic
```

### Problem: Real-time updates not received

**Symptoms:**

- WebSocket connected but no events

**Diagnosis:**

```bash
# Check if events are being published
docker compose logs api | grep "websocket"
docker compose logs api | grep "emit"
```

**Solution:**

```bash
# Verify subscription
# Client must send: { "type": "subscribe", "channel": "project:abc123" }

# Check Redis pub/sub
docker compose exec redis redis-cli PUBSUB CHANNELS
```

---

## Performance Issues

### Problem: Slow API responses

**Diagnosis:**

```bash
# Enable debug logging
LOG_LEVEL=debug docker compose up -d api

# Check slow queries
docker compose exec postgres psql -U yemaya << EOF
SELECT
  mean_exec_time,
  calls,
  query
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;
EOF
```

**Solutions:**

1. **Add database indexes:**

```sql
CREATE INDEX CONCURRENTLY idx_xxx ON table(column);
```

2. **Enable caching:**

```typescript
// Use Redis caching for expensive queries
const cached = await redis.get(`cache:${key}`);
if (cached) return JSON.parse(cached);
```

3. **Optimize queries:**

```typescript
// Use pagination
{ page: 1, limit: 20 }

// Select only needed fields
select: { id: true, name: true }

// Use proper joins
include: { relation: true }
```

### Problem: High memory usage

**Diagnosis:**

```bash
# Check container memory
docker stats

# Check Node.js heap
docker compose exec api node -e "console.log(process.memoryUsage())"
```

**Solution:**

```bash
# Increase memory limits
deploy:
  resources:
    limits:
      memory: 4G

# Enable garbage collection
NODE_OPTIONS="--max-old-space-size=4096"

# Check for memory leaks
# Use --inspect and Chrome DevTools
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js
```

### Problem: High CPU usage

**Diagnosis:**

```bash
# Check CPU usage
docker stats --format "table {{.Name}}\t{{.CPUPerc}}"

# Profile Node.js
docker compose exec api node --prof dist/index.js
```

**Solution:**

```bash
# Scale horizontally
docker compose up -d --scale api=4

# Optimize CPU-intensive operations
# Move to worker queue
```

---

## Docker Issues

### Problem: Container keeps restarting

**Diagnosis:**

```bash
# Check exit code
docker compose ps
docker inspect yemaya-api --format='{{.State.ExitCode}}'

# Check logs
docker compose logs --tail=100 api
```

**Common exit codes:**

- `0`: Normal exit
- `1`: Application error
- `137`: OOM killed (increase memory)
- `143`: SIGTERM (normal shutdown)

### Problem: Volumes not persisting

**Symptoms:**

- Data lost after restart

**Solution:**

```bash
# Check volume exists
docker volume ls | grep yemaya

# Inspect volume
docker volume inspect yemaya_postgres_data

# Ensure volume is mounted correctly
volumes:
  - postgres_data:/var/lib/postgresql/data  # Named volume
  # NOT - ./data:/var/lib/postgresql/data   # Bind mount (check permissions)
```

### Problem: Network issues between containers

**Diagnosis:**

```bash
# Check network
docker network ls
docker network inspect yemaya_backend

# Test connectivity
docker compose exec api ping postgres
docker compose exec api nc -zv postgres 5432
```

**Solution:**

```bash
# Ensure services are on same network
networks:
  - backend

# Recreate networks
docker compose down
docker network prune
docker compose up -d
```

---

## Kubernetes Issues

### Problem: Pod CrashLoopBackOff

**Diagnosis:**

```bash
# Check pod status
kubectl describe pod yemaya-api-xxx -n yemaya

# Check logs
kubectl logs yemaya-api-xxx -n yemaya --previous
```

**Common causes:**

- Missing secrets/configmaps
- Health check failing
- Application crash

**Solution:**

```bash
# Fix the underlying issue based on logs

# Verify secrets exist
kubectl get secrets -n yemaya

# Check configmap
kubectl get configmap -n yemaya
```

### Problem: Service not reachable

**Diagnosis:**

```bash
# Check service
kubectl get svc -n yemaya
kubectl describe svc yemaya-api -n yemaya

# Check endpoints
kubectl get endpoints yemaya-api -n yemaya

# Test from within cluster
kubectl run debug --image=curlimages/curl -it --rm -- curl yemaya-api:3000/health
```

### Problem: Ingress not working

**Diagnosis:**

```bash
# Check ingress
kubectl get ingress -n yemaya
kubectl describe ingress yemaya -n yemaya

# Check ingress controller logs
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller
```

**Solution:**

```bash
# Verify TLS secret exists
kubectl get secret yemaya-tls -n yemaya

# Check ingress class
kubectl get ingressclass
```

---

## Logging & Debugging

### Enable Debug Logging

```bash
# Set log level to debug
LOG_LEVEL=debug docker compose up -d api

# Enable pretty printing
LOG_PRETTY=true docker compose up -d api
```

### View Structured Logs

```bash
# Parse JSON logs
docker compose logs api | jq 'select(.level == "error")'

# Filter by correlation ID
docker compose logs api | jq 'select(.correlation_id == "req-abc123")'
```

### Remote Debugging

```bash
# Enable Node.js inspector
docker compose exec api node --inspect=0.0.0.0:9229 dist/index.js

# Connect Chrome DevTools
chrome://inspect

# Or VS Code
{
  "type": "node",
  "request": "attach",
  "port": 9229,
  "remoteRoot": "/app"
}
```

### Database Query Logging

```bash
# Enable query logging
docker compose exec postgres psql -U yemaya << EOF
ALTER SYSTEM SET log_statement = 'all';
ALTER SYSTEM SET log_min_duration_statement = 0;
SELECT pg_reload_conf();
EOF

# View query logs
docker compose logs postgres | grep "LOG:  statement:"
```

### Request Tracing

```bash
# All requests have correlation IDs
# Check X-Request-ID header in response

# Search logs by correlation ID
docker compose logs api | grep "req-abc123"
```

---

## Getting Help

If you can't resolve an issue:

1. **Check documentation:**
   - [API Reference](../api/README.md)
   - [Architecture Guide](../architecture/pipelines.md)
   - [Deployment Guide](./README.md)

2. **Search existing issues:**
   - https://github.com/yemaya/yemaya/issues

3. **Create a new issue with:**
   - Environment (OS, Docker version, Node version)
   - Steps to reproduce
   - Expected vs actual behavior
   - Relevant logs (sanitized of secrets)

4. **Community support:**
   - Discord: https://discord.gg/yemaya
   - Discussions: https://github.com/yemaya/yemaya/discussions

---

## Quick Reference

### Health Check URLs

| Service   | URL                                     | Expected                |
| --------- | --------------------------------------- | ----------------------- |
| API       | http://localhost:3000/health            | `{"status":"healthy"}`  |
| API Ready | http://localhost:3000/health/ready      | `{"status":"ready"}`    |
| MinIO     | http://localhost:9000/minio/health/live | HTTP 200                |
| Kibana    | http://localhost:5601/api/status        | `{"level":"available"}` |

### Log Locations

| Container     | Log Path                      |
| ------------- | ----------------------------- |
| API           | stdout (docker logs)          |
| Workers       | stdout (docker logs)          |
| PostgreSQL    | /var/lib/postgresql/data/log/ |
| Nginx         | /var/log/nginx/               |
| Elasticsearch | /var/log/elasticsearch/       |

### Environment Variables Reference

| Variable            | Required | Description                    |
| ------------------- | -------- | ------------------------------ |
| `DATABASE_URL`      | Yes      | PostgreSQL connection string   |
| `REDIS_URL`         | Yes      | Redis connection string        |
| `JWT_SECRET`        | Yes      | JWT signing secret (32+ chars) |
| `MINIO_*`           | Yes      | MinIO configuration            |
| `ANTHROPIC_API_KEY` | For AI   | Claude API key                 |
| `OPENAI_API_KEY`    | For AI   | OpenAI API key                 |
