# Debugging Guide

Comprehensive guide for debugging Lilith platform services during development.

## Table of Contents

- [IDE Setup](#ide-setup)
- [Node.js Debugging](#nodejs-debugging)
- [Log Analysis](#log-analysis)
- [Common Issues](#common-issues)
- [Performance Debugging](#performance-debugging)
- [Network Debugging](#network-debugging)
- [Database Debugging](#database-debugging)
- [Distributed Tracing](#distributed-tracing)

## IDE Setup

### VS Code Configuration

Create or update `.vscode/launch.json` for debugging services:

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Current Service",
      "program": "${workspaceFolder}/services/${input:serviceName}/src/server.ts",
      "preLaunchTask": "npm: build",
      "outFiles": [
        "${workspaceFolder}/services/${input:serviceName}/dist/**/*.js"
      ],
      "env": {
        "NODE_ENV": "development",
        "LOG_LEVEL": "debug"
      },
      "console": "integratedTerminal"
    },
    {
      "type": "node",
      "request": "attach",
      "name": "Attach to Running Service",
      "port": 9229,
      "restart": true,
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Jest Tests",
      "program": "${workspaceFolder}/node_modules/.bin/jest",
      "args": [
        "--runInBand",
        "--no-cache",
        "--testPathPattern=${fileBasename}"
      ],
      "cwd": "${workspaceFolder}/services/${input:serviceName}",
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ],
  "inputs": [
    {
      "id": "serviceName",
      "type": "pickString",
      "description": "Select service to debug",
      "options": [
        "auth",
        "bff",
        "content",
        "conversation",
        "media",
        "ai",
        "notification",
        "moderation"
      ]
    }
  ]
}
```

### Recommended VS Code Extensions

| Extension                                  | Purpose                   |
| ------------------------------------------ | ------------------------- |
| **ES7+ React/Redux/React-Native snippets** | Code snippets             |
| **ESLint**                                 | Linting integration       |
| **Prettier**                               | Code formatting           |
| **Docker**                                 | Container debugging       |
| **REST Client**                            | API testing               |
| **Thunder Client**                         | API requests from VS Code |
| **GitLens**                                | Git history and blame     |

## Node.js Debugging

### Starting a Service in Debug Mode

```bash
# Start with inspector
cd services/auth
node --inspect src/server.js

# Start with inspector on specific port
node --inspect=0.0.0.0:9229 src/server.js

# Break on first line
node --inspect-brk src/server.js
```

### Using npm Scripts

Add debug scripts to service `package.json`:

```json
{
  "scripts": {
    "debug": "node --inspect src/server.js",
    "debug:brk": "node --inspect-brk src/server.js",
    "debug:test": "node --inspect-brk node_modules/.bin/jest --runInBand"
  }
}
```

### Chrome DevTools Debugging

1. Start service with `--inspect` flag
2. Open Chrome and navigate to `chrome://inspect`
3. Click "Open dedicated DevTools for Node"
4. Set breakpoints and debug

### Debugging Tips

```javascript
// Add strategic breakpoints with debugger statement
async function processRequest(req) {
  debugger; // Execution pauses here when debugging
  const data = await fetchData(req.params.id);
  return transform(data);
}

// Conditional breakpoints in DevTools
// Right-click on breakpoint → "Edit breakpoint"
// Add condition: userId === 'user_123'
```

## Log Analysis

### Structured Logging

All services use structured JSON logging:

```javascript
// Example log output
{
  "timestamp": "2025-01-15T10:30:00.000Z",
  "level": "info",
  "service": "auth",
  "correlationId": "req_abc123",
  "message": "User authenticated",
  "userId": "user_456",
  "method": "password",
  "duration": 145
}
```

### Log Levels

| Level   | When to Use                              |
| ------- | ---------------------------------------- |
| `error` | Exceptions, failures requiring attention |
| `warn`  | Potential issues, deprecated usage       |
| `info`  | Normal operations, request completion    |
| `debug` | Detailed flow, variable values           |
| `trace` | Very verbose, method entry/exit          |

### Filtering Logs

```bash
# Filter by log level
LOG_LEVEL=debug npm start

# Filter by correlation ID
cat logs.json | jq 'select(.correlationId == "req_abc123")'

# Filter by service and time range
cat logs.json | jq 'select(.service == "auth" and .timestamp > "2025-01-15T10:00:00Z")'

# Find all errors
cat logs.json | jq 'select(.level == "error")'

# Get unique error messages
cat logs.json | jq 'select(.level == "error") | .message' | sort | uniq -c | sort -rn
```

### Real-time Log Watching

```bash
# Watch service logs in development
npm run dev 2>&1 | jq -r '.message'

# Docker logs with follow
docker logs -f lilith-auth-1 | jq '.'

# Kubernetes logs
kubectl logs -f deploy/auth -n development | jq '.'

# Multiple pods
kubectl logs -f -l app=auth -n development --all-containers
```

## Common Issues

### Authentication Failures

**Symptom:** 401 Unauthorized errors

```bash
# Check JWT configuration
curl -X POST http://localhost:3003/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"test@example.com","password":"password123"}'
```

**Debug steps:**

1. Verify JWT secret is set: `echo $JWT_SECRET`
2. Check token expiry configuration
3. Inspect token with jwt.io
4. Check clock skew between services

```javascript
// Decode and inspect token
const jwt = require('jsonwebtoken');
const decoded = jwt.decode(token, { complete: true });
console.log('Header:', decoded.header);
console.log('Payload:', decoded.payload);
console.log('Expires:', new Date(decoded.payload.exp * 1000));
```

### Database Connection Issues

**Symptom:** Connection timeouts, pool exhaustion

```bash
# Check PostgreSQL connection
psql -h localhost -p 5433 -U lilith -d lilith_dev -c "SELECT 1"

# Check connection pool status
psql -c "SELECT * FROM pg_stat_activity WHERE datname = 'lilith_dev'"

# Check for long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
         FROM pg_stat_activity
         WHERE state = 'active'
         ORDER BY duration DESC"
```

**Debug steps:**

1. Verify connection string in environment
2. Check pool size configuration
3. Look for connection leaks (missing `.release()` calls)
4. Check for deadlocks

### Redis Connection Issues

**Symptom:** Cache misses, session failures

```bash
# Test Redis connection
redis-cli -p 6380 ping

# Check memory usage
redis-cli -p 6380 info memory

# Monitor commands in real-time
redis-cli -p 6380 monitor

# Check slow log
redis-cli -p 6380 slowlog get 10
```

### Service Communication Failures

**Symptom:** Inter-service calls failing

```bash
# Check service health endpoints
curl http://localhost:3000/health  # BFF
curl http://localhost:3003/health  # Auth
curl http://localhost:3004/health  # Content

# Check service discovery
docker compose ps
kubectl get pods -n development

# Test internal routing
docker exec lilith-bff-1 curl http://auth:3003/health
```

### Memory Issues

**Symptom:** OOM errors, increasing memory usage

```bash
# Monitor Node.js memory
node --expose-gc src/server.js

# In another terminal, trigger GC and get heap stats
kill -USR1 $(pgrep -f "node.*server.js")

# Generate heap snapshot
node --heapsnapshot-signal=SIGUSR2 src/server.js
kill -USR2 $(pgrep -f "node.*server.js")
```

```javascript
// Add memory monitoring
setInterval(() => {
  const used = process.memoryUsage();
  console.log('Memory:', {
    heapUsed: Math.round(used.heapUsed / 1024 / 1024) + 'MB',
    heapTotal: Math.round(used.heapTotal / 1024 / 1024) + 'MB',
    external: Math.round(used.external / 1024 / 1024) + 'MB',
  });
}, 10000);
```

## Performance Debugging

### Identifying Slow Endpoints

```bash
# Add timing middleware
app.addHook('onRequest', async (request) => {
  request.startTime = process.hrtime.bigint();
});

app.addHook('onResponse', async (request, reply) => {
  const duration = Number(process.hrtime.bigint() - request.startTime) / 1e6;
  if (duration > 1000) {
    console.warn('Slow request', {
      method: request.method,
      url: request.url,
      duration: `${duration.toFixed(2)}ms`
    });
  }
});
```

### CPU Profiling

```bash
# Start with profiling
node --prof src/server.js

# Generate readable output
node --prof-process isolate-*.log > profile.txt

# Use 0x for flame graphs
npx 0x src/server.js
```

### Memory Profiling

```bash
# Generate heap snapshot on demand
node --heapsnapshot-signal=SIGUSR2 src/server.js

# Send signal to generate snapshot
kill -USR2 <PID>

# Analyze in Chrome DevTools (Memory tab)
```

### Database Query Performance

```javascript
// Log slow queries
const queryLogger = (sql, params, duration) => {
  if (duration > 100) {
    console.warn('Slow query', { sql, params, duration: `${duration}ms` });
  }
};

// PostgreSQL with pg
const pool = new Pool({
  log: queryLogger,
});

// Or add EXPLAIN ANALYZE
const result = await pool.query(
  `
  EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)
  ${originalQuery}
`,
  params
);
console.log(JSON.stringify(result.rows[0], null, 2));
```

## Network Debugging

### Request Tracing

```bash
# Trace HTTP requests with curl
curl -v http://localhost:3000/v1/chat

# Trace with timing
curl -w "@curl-format.txt" -o /dev/null -s http://localhost:3000/health
```

Create `curl-format.txt`:

```
     time_namelookup:  %{time_namelookup}s\n
        time_connect:  %{time_connect}s\n
     time_appconnect:  %{time_appconnect}s\n
    time_pretransfer:  %{time_pretransfer}s\n
       time_redirect:  %{time_redirect}s\n
  time_starttransfer:  %{time_starttransfer}s\n
                     ----------\n
          time_total:  %{time_total}s\n
```

### DNS Resolution

```bash
# Check DNS resolution
nslookup auth.lilith.local
dig auth.lilith.local

# Check /etc/hosts
cat /etc/hosts | grep lilith

# In Docker, check network
docker network inspect lilith_default
```

### SSL/TLS Issues

```bash
# Check certificate
openssl s_client -connect api.lilith.app:443 -servername api.lilith.app

# Verify certificate chain
openssl s_client -connect api.lilith.app:443 -showcerts

# Check expiry
echo | openssl s_client -connect api.lilith.app:443 2>/dev/null | openssl x509 -noout -dates
```

## Database Debugging

### PostgreSQL

```bash
# Connect to database
psql -h localhost -p 5433 -U lilith -d lilith_dev

# Common debug queries
# Active connections
SELECT * FROM pg_stat_activity WHERE state = 'active';

# Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

# Index usage
SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;

# Missing indexes (tables with sequential scans)
SELECT schemaname, relname, seq_scan, idx_scan
FROM pg_stat_user_tables
WHERE seq_scan > idx_scan
ORDER BY seq_scan DESC;

# Lock monitoring
SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
WHERE NOT blocked_locks.granted;
```

### Redis

```bash
# Connect to Redis
redis-cli -p 6380

# Debug commands
DEBUG OBJECT <key>
MEMORY USAGE <key>
OBJECT ENCODING <key>

# Find large keys
redis-cli -p 6380 --bigkeys

# Scan for patterns
redis-cli -p 6380 --scan --pattern "session:*" | head -20

# Check TTL
redis-cli -p 6380 TTL <key>
```

## Distributed Tracing

### Correlation IDs

All requests should carry a correlation ID through the system:

```javascript
import { generateUUID } from '@lilith/core';

// Middleware to add correlation ID
app.addHook('onRequest', async (request) => {
  request.correlationId = request.headers['x-correlation-id'] || generateUUID();
  request.log = request.log.child({ correlationId: request.correlationId });
});

// Pass to downstream services
const response = await fetch('http://content:3004/v1/courses', {
  headers: {
    'X-Correlation-ID': request.correlationId,
  },
});
```

### Jaeger Tracing

Access Jaeger UI at `http://localhost:16686` in development.

```javascript
// Add spans for operations
const span = tracer.startSpan('database.query', { childOf: parentSpan });
try {
  const result = await db.query(sql, params);
  span.setTag('rows', result.rowCount);
  return result;
} catch (error) {
  span.setTag('error', true);
  span.log({ event: 'error', message: error.message });
  throw error;
} finally {
  span.finish();
}
```

### Finding Traces

```bash
# Query Jaeger API for traces
curl "http://localhost:16686/api/traces?service=bff&limit=20"

# Find traces by correlation ID
curl "http://localhost:16686/api/traces?tags=%7B%22correlationId%22%3A%22req_abc123%22%7D"
```

## Quick Reference

### Environment Variables for Debugging

```bash
# Enable verbose logging
LOG_LEVEL=debug

# Enable Node.js inspector
NODE_OPTIONS="--inspect"

# Enable detailed error stacks
NODE_ENV=development

# PostgreSQL query logging
DEBUG=pg:*

# Redis command logging
DEBUG=redis:*
```

### Useful Commands

```bash
# Service health check
curl -s http://localhost:3000/health | jq '.'

# View service logs
docker compose logs -f auth

# Enter container shell
docker compose exec auth sh

# Run command in container
docker compose exec auth npm test

# Check resource usage
docker stats

# Restart specific service
docker compose restart auth

# View environment variables
docker compose exec auth env | sort
```

---

## Related Documentation

- [Local Development Setup](./local-development-setup.md)
- [Testing Strategy](../testing-strategy.md)
- [Incident Response Runbook](../runbooks/incidents/incident-response.md)
- [Troubleshooting Guide](../runbooks/incidents/troubleshooting.md)
