Overview#
This guide covers how to run, monitor, debug, and maintain the Lilith platform in production and development environments.
Table of Contents#
- Local Development
- Deployment
- Monitoring
- Alerting
- Incident Response
- Database Operations
- Troubleshooting
- Runbook Quick Reference
Local Development#
Prerequisites#
- Node.js 18+
- Docker and Docker Compose
- PostgreSQL 15+
- Redis 7+
Quick Start#
bash
# Clone and install
git clone https://github.com/lilith/lilith
cd lilith
npm install
# Start infrastructure
docker compose up -d postgres redis kafka milvus
# Seed development data
npm run seed:dev
# Start services
npm run deploy
# Verify health
npm run health:wait
Service Ports#
| Service | Port | Description |
|---|---|---|
| BFF | 8000 | API Gateway |
| Auth | 8001 | Authentication |
| Content | 8002 | Content Management |
| Conversation | 8003 | Chat/Messages |
| AI | 8004 | AI Orchestration |
| Media | 8005 | Media/Assets |
| TTS | 8011 | Text-to-Speech |
| STT | 8010 | Speech-to-Text |
Development Commands#
bash
# View logs
npm run logs # All services
npm run logs:errors # Errors only
npm run logs -- --service ai # Specific service
# Run tests
npm test # All tests
npm run test:e2e # E2E tests
npm run test:coverage # With coverage
# Code quality
npm run lint # Lint check
npm run format # Format code
npm run typecheck # Type check
Deployment#
Environments#
| Environment | URL | Purpose |
|---|---|---|
| Local | localhost:8000 | Development |
| Staging | staging.api.lilith.app | Pre-production |
| Production | api.lilith.app | Live users |
Kubernetes Deployment#
bash
# Deploy to staging
kubectl apply -k deploy/kubernetes/overlays/staging
# Deploy to production
kubectl apply -k deploy/kubernetes/overlays/production
# Check rollout status
kubectl rollout status deployment/ai-service -n lilith
Helm Deployment#
bash
# Install/upgrade
helm upgrade --install lilith deploy/helm/lilith-service \
--namespace lilith \
--values deploy/helm/values-production.yaml
# Rollback
helm rollback lilith 1 -n lilith
ArgoCD (GitOps)#
bash
# Sync application
argocd app sync lilith-app-of-apps
# Check status
argocd app get lilith-app-of-apps
# Manual sync (emergency)
argocd app sync lilith-app-of-apps --force
Rolling Updates#
- Update image tag in values file
- Commit to main branch
- ArgoCD auto-syncs (or manual sync)
- Monitor rollout health
- Rollback if issues detected
Monitoring#
Key Dashboards#
| Dashboard | URL | Purpose |
|---|---|---|
| Service Health | /grafana/d/services | Overall system health |
| AI Metrics | /grafana/d/ai | LLM latency, tokens |
| TTS/STT | /grafana/d/speech | Audio service metrics |
| User Activity | /grafana/d/users | DAU, sessions |
Key Metrics#
Latency (SLO Targets)#
| Metric | p50 Target | p95 Target |
|---|---|---|
| Chat Response | < 2s | < 4s |
| TTS First Byte | < 600ms | < 1s |
| API Endpoints | < 200ms | < 500ms |
Error Rates#
| Metric | Target |
|---|---|
| Error Rate | < 0.1% |
| 5xx Rate | < 0.01% |
| Circuit Breaker Trips | < 5/hour |
Health Checks#
bash
# Service health endpoint
curl https://api.lilith.app/v1/health
# Kubernetes pod health
kubectl get pods -n lilith -o wide
# Service mesh health
istioctl proxy-status
Logging#
Log Aggregation#
Logs flow to ELK stack:
text
Service → Fluentd → Elasticsearch → Kibana
Log Queries#
bash
# Kibana: Find errors for AI service
service:ai AND level:error AND @timestamp:[now-1h TO now]
# Find slow requests
response_time_ms:>2000 AND service:bff
# Trace specific request
correlation_id:"abc123"
Tracing#
Jaeger UI#
Access at: /jaeger
Trace a request:
- Get correlation ID from logs or response header
- Search in Jaeger by trace ID
- View span waterfall
Alerting#
Alert Channels#
| Severity | Channel | Response Time |
|---|---|---|
| Critical | PagerDuty | < 15 min |
| High | Slack #alerts | < 1 hour |
| Medium | < 4 hours | |
| Low | Ticket | Next business day |
Critical Alerts#
| Alert | Condition | Action |
|---|---|---|
| Service Down | Pod restarts > 3 in 5min | Check logs, restart |
| High Error Rate | 5xx > 1% for 5min | Investigate, rollback |
| Database Unreachable | Connection failures | Check DB, failover |
| TTS Latency | p95 > 2s for 10min | Scale up, check GPU |
Alert Response#
- Acknowledge alert in PagerDuty
- Check dashboard for scope
- Review recent deployments
- Check logs for root cause
- Apply mitigation
- Update incident channel
- Post-incident review
Incident Response#
Severity Levels#
| Level | Definition | Response |
|---|---|---|
| SEV-1 | Complete outage | All hands, < 15min |
| SEV-2 | Major feature down | On-call + backup, < 30min |
| SEV-3 | Degraded performance | On-call, < 1hr |
| SEV-4 | Minor issue | Business hours |
Incident Process#
- Detect: Alert fires or user report
- Triage: Assess severity, notify stakeholders
- Mitigate: Apply quick fix (rollback, scale, restart)
- Communicate: Status page, Slack updates
- Resolve: Fix root cause
- Review: Post-incident analysis
Communication Templates#
Initial:
We are investigating reports of [issue]. Updates to follow.
Mitigation:
We have identified the issue as [cause] and are applying [fix]. Service should recover within [time].
Resolution:
The issue has been resolved. [Summary]. We will conduct a post-incident review.
Runbook Links#
Database Operations#
PostgreSQL#
Connection#
bash
# Production (via bastion)
psql -h db.lilith.internal -U lilith -d lilith
# Local
psql -h localhost -U postgres -d lilith
Common Operations#
sql
-- Check connections
SELECT count(*) FROM pg_stat_activity;
-- Slow queries
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC LIMIT 10;
-- Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
Backup/Restore#
bash
# Backup
pg_dump -h db.lilith.internal -U lilith lilith > backup.sql
# Restore
psql -h db.lilith.internal -U lilith lilith < backup.sql
Redis#
bash
# Connect
redis-cli -h redis.lilith.internal
# Memory usage
INFO memory
# Key patterns
KEYS "session:*" | wc -l
Milvus (Vector DB)#
bash
# Health check
curl http://milvus.lilith.internal:19530/v1/health
# Collection stats
curl http://milvus.lilith.internal:19530/v1/collections/content_embeddings
Troubleshooting#
Common Issues#
High Latency#
- Check service metrics in Grafana
- Review traces in Jaeger
- Check database query times
- Verify network latency (service mesh)
- Check for resource contention
bash
# Check pod resources
kubectl top pods -n lilith
# Check network
istioctl proxy-status
Pod CrashLoopBackOff#
- Check pod logs
- Verify environment variables
- Check resource limits
- Verify health check endpoints
- Check dependencies (DB, cache)
bash
kubectl logs -n lilith <pod-name> --previous
kubectl describe pod -n lilith <pod-name>
Database Connection Issues#
- Check connection pool usage
- Verify credentials
- Check network policies
- Test connectivity from pod
bash
# Test from pod
kubectl exec -it <pod> -n lilith -- nc -zv postgres 5432
TTS/STT Failures#
- Check GPU availability
- Verify model loading
- Check queue depth
- Review error logs
bash
# GPU status
nvidia-smi
# TTS queue
curl http://tts:8011/v1/queue/status
Debug Mode#
bash
# Enable debug logging (temporary)
kubectl set env deployment/ai-service LOG_LEVEL=debug -n lilith
# Revert
kubectl set env deployment/ai-service LOG_LEVEL=info -n lilith
Runbook Quick Reference#
Emergency Procedures#
| Scenario | Command/Action |
|---|---|
| Service unresponsive | kubectl rollout restart deployment/<svc> -n lilith |
| Database overload | Scale read replicas, kill long queries |
| High error rate | Rollback: helm rollback lilith |
| DDoS detected | Enable WAF rules, scale up |
| Secrets compromised | Rotate keys via Vault |
Scaling#
bash
# Manual scale
kubectl scale deployment/ai-service --replicas=10 -n lilith
# HPA adjustment (temporary)
kubectl patch hpa ai-service -n lilith -p '{"spec":{"maxReplicas":50}}'
Maintenance Windows#
| Task | Frequency | Impact |
|---|---|---|
| Database backup | Daily | None |
| Index rebuild | Weekly | Minor latency |
| Certificate renewal | Quarterly | Rolling update |
| Dependency updates | Monthly | Scheduled downtime |