# Lilith Operations Guide

## Overview

This guide covers how to run, monitor, debug, and maintain the Lilith platform
in production and development environments.

## Table of Contents

1. [Local Development](#local-development)
2. [Deployment](#deployment)
3. [Monitoring](#monitoring)
4. [Alerting](#alerting)
5. [Incident Response](#incident-response)
6. [Database Operations](#database-operations)
7. [Troubleshooting](#troubleshooting)
8. [Runbook Quick Reference](#runbook-quick-reference)

---

## Local Development

### Prerequisites

- Node.js 18+
- Docker and Docker Compose
- PostgreSQL 15+
- Redis 7+

### Quick Start

```bash
# Clone and install
git clone https://github.com/lilith/lilith
cd lilith
npm install

# Start infrastructure
docker compose up -d postgres redis kafka milvus

# Seed development data
npm run seed:dev

# Start services
npm run deploy

# Verify health
npm run health:wait
```

### Service Ports

| Service      | Port | Description        |
| ------------ | ---- | ------------------ |
| BFF          | 8000 | API Gateway        |
| Auth         | 8001 | Authentication     |
| Content      | 8002 | Content Management |
| Conversation | 8003 | Chat/Messages      |
| AI           | 8004 | AI Orchestration   |
| Media        | 8005 | Media/Assets       |
| TTS          | 8011 | Text-to-Speech     |
| STT          | 8010 | Speech-to-Text     |

### Development Commands

```bash
# View logs
npm run logs              # All services
npm run logs:errors       # Errors only
npm run logs -- --service ai  # Specific service

# Run tests
npm test                  # All tests
npm run test:e2e          # E2E tests
npm run test:coverage     # With coverage

# Code quality
npm run lint              # Lint check
npm run format            # Format code
npm run typecheck         # Type check
```

---

## Deployment

### Environments

| Environment | URL                    | Purpose        |
| ----------- | ---------------------- | -------------- |
| Local       | localhost:8000         | Development    |
| Staging     | staging.api.lilith.app | Pre-production |
| Production  | api.lilith.app         | Live users     |

### Kubernetes Deployment

```bash
# Deploy to staging
kubectl apply -k deploy/kubernetes/overlays/staging

# Deploy to production
kubectl apply -k deploy/kubernetes/overlays/production

# Check rollout status
kubectl rollout status deployment/ai-service -n lilith
```

### Helm Deployment

```bash
# Install/upgrade
helm upgrade --install lilith deploy/helm/lilith-service \
  --namespace lilith \
  --values deploy/helm/values-production.yaml

# Rollback
helm rollback lilith 1 -n lilith
```

### ArgoCD (GitOps)

```bash
# Sync application
argocd app sync lilith-app-of-apps

# Check status
argocd app get lilith-app-of-apps

# Manual sync (emergency)
argocd app sync lilith-app-of-apps --force
```

### Rolling Updates

1. Update image tag in values file
2. Commit to main branch
3. ArgoCD auto-syncs (or manual sync)
4. Monitor rollout health
5. Rollback if issues detected

---

## Monitoring

### Key Dashboards

| Dashboard      | URL                 | Purpose               |
| -------------- | ------------------- | --------------------- |
| Service Health | /grafana/d/services | Overall system health |
| AI Metrics     | /grafana/d/ai       | LLM latency, tokens   |
| TTS/STT        | /grafana/d/speech   | Audio service metrics |
| User Activity  | /grafana/d/users    | DAU, sessions         |

### Key Metrics

#### Latency (SLO Targets)

| Metric         | p50 Target | p95 Target |
| -------------- | ---------- | ---------- |
| Chat Response  | < 2s       | < 4s       |
| TTS First Byte | < 600ms    | < 1s       |
| API Endpoints  | < 200ms    | < 500ms    |

#### Error Rates

| Metric                | Target   |
| --------------------- | -------- |
| Error Rate            | < 0.1%   |
| 5xx Rate              | < 0.01%  |
| Circuit Breaker Trips | < 5/hour |

### Health Checks

```bash
# Service health endpoint
curl https://api.lilith.app/v1/health

# Kubernetes pod health
kubectl get pods -n lilith -o wide

# Service mesh health
istioctl proxy-status
```

### Logging

#### Log Aggregation

Logs flow to ELK stack:

```
Service → Fluentd → Elasticsearch → Kibana
```

#### Log Queries

```bash
# Kibana: Find errors for AI service
service:ai AND level:error AND @timestamp:[now-1h TO now]

# Find slow requests
response_time_ms:>2000 AND service:bff

# Trace specific request
correlation_id:"abc123"
```

### Tracing

#### Jaeger UI

Access at: `/jaeger`

Trace a request:

1. Get correlation ID from logs or response header
2. Search in Jaeger by trace ID
3. View span waterfall

---

## Alerting

### Alert Channels

| Severity | Channel       | Response Time     |
| -------- | ------------- | ----------------- |
| Critical | PagerDuty     | < 15 min          |
| High     | Slack #alerts | < 1 hour          |
| Medium   | Email         | < 4 hours         |
| Low      | Ticket        | Next business day |

### Critical Alerts

| Alert                | Condition                | Action                |
| -------------------- | ------------------------ | --------------------- |
| Service Down         | Pod restarts > 3 in 5min | Check logs, restart   |
| High Error Rate      | 5xx > 1% for 5min        | Investigate, rollback |
| Database Unreachable | Connection failures      | Check DB, failover    |
| TTS Latency          | p95 > 2s for 10min       | Scale up, check GPU   |

### Alert Response

1. Acknowledge alert in PagerDuty
2. Check dashboard for scope
3. Review recent deployments
4. Check logs for root cause
5. Apply mitigation
6. Update incident channel
7. Post-incident review

---

## Incident Response

### Severity Levels

| Level | Definition           | Response                  |
| ----- | -------------------- | ------------------------- |
| SEV-1 | Complete outage      | All hands, < 15min        |
| SEV-2 | Major feature down   | On-call + backup, < 30min |
| SEV-3 | Degraded performance | On-call, < 1hr            |
| SEV-4 | Minor issue          | Business hours            |

### Incident Process

1. **Detect**: Alert fires or user report
2. **Triage**: Assess severity, notify stakeholders
3. **Mitigate**: Apply quick fix (rollback, scale, restart)
4. **Communicate**: Status page, Slack updates
5. **Resolve**: Fix root cause
6. **Review**: Post-incident analysis

### Communication Templates

**Initial**:

> We are investigating reports of [issue]. Updates to follow.

**Mitigation**:

> We have identified the issue as [cause] and are applying [fix]. Service should
> recover within [time].

**Resolution**:

> The issue has been resolved. [Summary]. We will conduct a post-incident
> review.

### Runbook Links

- [Incident Response](../docs/runbooks/incidents/incident-response.md)
- [Database Recovery](../docs/runbooks/disaster-recovery/database-recovery.md)
- [Service Recovery](../docs/runbooks/disaster-recovery/service-recovery.md)

---

## Database Operations

### PostgreSQL

#### Connection

```bash
# Production (via bastion)
psql -h db.lilith.internal -U lilith -d lilith

# Local
psql -h localhost -U postgres -d lilith
```

#### Common Operations

```sql
-- Check connections
SELECT count(*) FROM pg_stat_activity;

-- Slow queries
SELECT query, calls, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC LIMIT 10;

-- Table sizes
SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
FROM pg_catalog.pg_statio_user_tables
ORDER BY pg_total_relation_size(relid) DESC;
```

#### Backup/Restore

```bash
# Backup
pg_dump -h db.lilith.internal -U lilith lilith > backup.sql

# Restore
psql -h db.lilith.internal -U lilith lilith < backup.sql
```

### Redis

```bash
# Connect
redis-cli -h redis.lilith.internal

# Memory usage
INFO memory

# Key patterns
KEYS "session:*" | wc -l
```

### Milvus (Vector DB)

```bash
# Health check
curl http://milvus.lilith.internal:19530/v1/health

# Collection stats
curl http://milvus.lilith.internal:19530/v1/collections/content_embeddings
```

---

## Troubleshooting

### Common Issues

#### High Latency

1. Check service metrics in Grafana
2. Review traces in Jaeger
3. Check database query times
4. Verify network latency (service mesh)
5. Check for resource contention

```bash
# Check pod resources
kubectl top pods -n lilith

# Check network
istioctl proxy-status
```

#### Pod CrashLoopBackOff

1. Check pod logs
2. Verify environment variables
3. Check resource limits
4. Verify health check endpoints
5. Check dependencies (DB, cache)

```bash
kubectl logs -n lilith <pod-name> --previous
kubectl describe pod -n lilith <pod-name>
```

#### Database Connection Issues

1. Check connection pool usage
2. Verify credentials
3. Check network policies
4. Test connectivity from pod

```bash
# Test from pod
kubectl exec -it <pod> -n lilith -- nc -zv postgres 5432
```

#### TTS/STT Failures

1. Check GPU availability
2. Verify model loading
3. Check queue depth
4. Review error logs

```bash
# GPU status
nvidia-smi

# TTS queue
curl http://tts:8011/v1/queue/status
```

### Debug Mode

```bash
# Enable debug logging (temporary)
kubectl set env deployment/ai-service LOG_LEVEL=debug -n lilith

# Revert
kubectl set env deployment/ai-service LOG_LEVEL=info -n lilith
```

---

## Runbook Quick Reference

### Emergency Procedures

| Scenario             | Command/Action                                       |
| -------------------- | ---------------------------------------------------- |
| Service unresponsive | `kubectl rollout restart deployment/<svc> -n lilith` |
| Database overload    | Scale read replicas, kill long queries               |
| High error rate      | Rollback: `helm rollback lilith`                     |
| DDoS detected        | Enable WAF rules, scale up                           |
| Secrets compromised  | Rotate keys via Vault                                |

### Scaling

```bash
# Manual scale
kubectl scale deployment/ai-service --replicas=10 -n lilith

# HPA adjustment (temporary)
kubectl patch hpa ai-service -n lilith -p '{"spec":{"maxReplicas":50}}'
```

### Maintenance Windows

| Task                | Frequency | Impact             |
| ------------------- | --------- | ------------------ |
| Database backup     | Daily     | None               |
| Index rebuild       | Weekly    | Minor latency      |
| Certificate renewal | Quarterly | Rolling update     |
| Dependency updates  | Monthly   | Scheduled downtime |

---

## Related Documentation

- [Architecture](./architecture.md)
- [Runbooks Directory](../docs/runbooks/)
- [Infrastructure Setup](../docs/INFRASTRUCTURE_SETUP.md)
- [Local Development Guide](../docs/LOCAL_DEVELOPMENT_GUIDE.md)
