# Health Monitoring and Automatic Restart Policies

## Overview

This document describes the comprehensive health monitoring and automatic
restart system implemented for the Lilith platform. The system provides
real-time monitoring, automated recovery, and detailed health reporting for all
services.

## Architecture

### Components

1. **Docker Health Checks**: Native Docker health checks for all services
2. **Restart Policies**: Automated service recovery based on health status
3. **Resource Limits**: Memory and CPU constraints to prevent resource
   exhaustion
4. **Monitoring Scripts**: Custom scripts for comprehensive health analysis
5. **Alerting**: Critical service notifications and error detection

### Configuration Files

- `docker-compose.health.yml` - Health checks and restart policies overlay
- `scripts/health-monitor.sh` - Comprehensive monitoring script
- `docker-compose.dev.yml` - Base configuration with health checks

## Health Check Configuration

### Application Services

All application services implement standardized health checks:

```yaml
healthcheck:
  test: ['CMD', 'curl', '-f', 'http://localhost:8080/health']
  interval: 30s # Check every 30 seconds
  timeout: 10s # Timeout after 10 seconds
  retries: 3 # Allow 3 failures before marking unhealthy
  start_period: 60s # Grace period for startup
```

### Restart Policies

Services use different restart strategies based on criticality:

```yaml
restart: unless-stopped # Restart unless manually stopped
```

### Resource Limits

Memory constraints prevent resource exhaustion:

```yaml
deploy:
  resources:
    limits:
      memory: 512M # Maximum memory
    reservations:
      memory: 256M # Guaranteed memory
```

## Service Categories

### Core Services (Critical)

- **bff**: Backend for Frontend
- **auth**: Authentication Service
- **conversation**: Conversation Management
- **ai**: AI/LLM Service

**Memory Allocation**: 512MB limit, 256MB reserved

### Resource-Intensive Services

- **tts**: Text-to-Speech (2GB limit, 1GB reserved)
- **stt**: Speech-to-Text (2GB limit, 1GB reserved)
- **ai**: AI Service with models (1GB limit, 512MB reserved)
- **rag**: RAG/Vector Service (1GB limit, 512MB reserved)

### Database Services

- **postgres**: PostgreSQL Database (2GB limit, 1GB reserved)
- **elasticsearch**: Search Engine (2GB limit, 1GB reserved)
- **neo4j**: Graph Database (2GB limit, 1GB reserved)
- **milvus**: Vector Database (4GB limit, 2GB reserved)

## Monitoring Scripts

### health-monitor.sh

Comprehensive monitoring script with multiple modes:

```bash
# Check all services
./scripts/health-monitor.sh all

# Watch continuously (every 30 seconds)
./scripts/health-monitor.sh watch

# Watch with custom interval
./scripts/health-monitor.sh watch 60

# Critical service alerts
./scripts/health-monitor.sh alerts

# Generate health report
./scripts/health-monitor.sh report

# Check specific service categories
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases
./scripts/health-monitor.sh utilities
```

### Features

1. **Service Health Status**: Real-time health checks for all services
2. **Response Time Monitoring**: Latency measurements for service endpoints
3. **Resource Usage**: CPU and memory consumption tracking
4. **Error Detection**: Automatic error log analysis
5. **Container Status**: Docker container state monitoring
6. **System Health**: Overall platform status summary

## Health Endpoints

### Standard Health Response

All services implement a standardized health endpoint:

```javascript
// GET /health
{
  "status": "healthy",
  "timestamp": "2025-01-06T12:00:00Z",
  "service": "service-name",
  "version": "1.0.0",
  "uptime": 3600,
  "dependencies": {
    "database": "healthy",
    "redis": "healthy"
  },
  "checks": {
    "memory": "ok",
    "disk": "ok",
    "external_apis": "ok"
  }
}
```

### Service-Specific Endpoints

- **AI Service**: `/v1/health` - Provider status and model availability
- **Database Services**: Database-specific health checks
- **Utilities**: Service-appropriate health endpoints

## Restart Strategies

### Automatic Restart

Services restart automatically when:

- Health check fails 3 consecutive times
- Container crashes or exits unexpectedly
- Resource limits exceeded

### Manual Recovery

```bash
# Restart specific service
docker compose -f docker-compose.dev.yml restart bff

# Restart all services
docker compose -f docker-compose.dev.yml restart

# Force recreation
docker compose -f docker-compose.dev.yml up -d --force-recreate
```

### Dependency Management

Services with dependencies wait for upstream services:

```yaml
depends_on:
  postgres:
    condition: service_healthy
  redis:
    condition: service_healthy
```

## Monitoring and Alerting

### Critical Service Monitoring

Core services are monitored for immediate alerts:

- BFF (Backend for Frontend)
- Authentication Service
- Conversation Service
- AI Service

### Alert Conditions

- Service health check failures
- High error rates in logs
- Resource exhaustion
- Dependency failures
- Response time degradation

### Log Monitoring

Automatic detection of error patterns:

- Error keywords: "error", "exception", "failed"
- Stack traces
- HTTP 5xx responses
- Database connection failures

## Performance Monitoring

### Response Time Tracking

Health checks measure response times:

- **Normal**: < 1000ms
- **Warning**: 1000-5000ms
- **Critical**: > 5000ms

### Resource Usage Monitoring

- **CPU Usage**: Container CPU utilization
- **Memory Usage**: Container memory consumption
- **Disk Usage**: Available disk space
- **Network I/O**: Network traffic patterns

## Maintenance and Troubleshooting

### Common Issues

1. **Services Not Starting**

   ```bash
   # Check health status
   ./scripts/health-monitor.sh all

   # View logs
   docker logs lilith-bff-1
   ```

2. **High Memory Usage**

   ```bash
   # Check resource usage
   ./scripts/health-monitor.sh docker

   # Restart memory-intensive services
   docker compose restart ai rag tts stt
   ```

3. **Database Connection Issues**

   ```bash
   # Check database health
   ./scripts/health-monitor.sh databases

   # Verify database connectivity
   docker exec lilith-postgres-1 pg_isready -U lilith -d lilith_dev
   ```

### Maintenance Tasks

1. **Health Check Tuning**
   - Adjust intervals for different service types
   - Customize timeout values based on service characteristics
   - Optimize start periods for slower services

2. **Resource Limit Optimization**
   - Monitor actual resource usage patterns
   - Adjust memory limits based on requirements
   - Scale resources for production workloads

3. **Alert Configuration**
   - Configure notification channels
   - Set up alert escalation policies
   - Customize alert thresholds

## Production Considerations

### High Availability

- **Service Redundancy**: Multiple instances for critical services
- **Load Balancing**: Health check-based routing
- **Graceful Degradation**: Fallback mechanisms for service failures

### Monitoring at Scale

- **Metrics Collection**: Prometheus/Grafana integration
- **Log Aggregation**: Centralized logging with ELK stack
- **Distributed Tracing**: Jaeger for request tracing
- **Alert Management**: PagerDuty/Slack integrations

### Disaster Recovery

- **Health Check Backup**: Secondary monitoring systems
- **Automated Failover**: Health-based traffic routing
- **Data Backup**: Automated database backups
- **Recovery Procedures**: Documented recovery playbooks

## Usage Examples

### Development Environment

```bash
# Start with health monitoring
docker compose -f docker-compose.dev.yml -f docker-compose.health.yml up -d

# Monitor health
./scripts/health-monitor.sh watch

# Check specific services
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases
```

### Test Environment

```bash
# Start test environment with health checks
docker compose -f docker-compose.dev.yml -f docker-compose.test.yml -f docker-compose.health.yml up -d

# Run health checks with alerts
./scripts/health-monitor.sh alerts
```

### Production Monitoring

```bash
# Continuous monitoring
nohup ./scripts/health-monitor.sh watch 60 > health.log 2>&1 &

# Generate daily reports
./scripts/health-monitor.sh report > daily-health-$(date +%Y%m%d).txt
```

## Best Practices

1. **Regular Health Checks**: Monitor services continuously
2. **Proactive Maintenance**: Address issues before they become critical
3. **Resource Planning**: Monitor resource usage trends
4. **Documentation**: Keep health configurations updated
5. **Testing**: Regularly test recovery procedures
6. **Alert Tuning**: Balance sensitivity and false positives
7. **Performance Monitoring**: Track response time trends
8. **Capacity Planning**: Scale resources based on monitoring data
