Eve library

Health Monitoring and Automatic Restart Policies

Guide

Revision 1

On this page

Overview#

This document describes the comprehensive health monitoring and automatic restart system implemented for the Lilith platform. The system provides real-time monitoring, automated recovery, and detailed health reporting for all services.

Architecture#

Components#

  1. Docker Health Checks: Native Docker health checks for all services
  2. Restart Policies: Automated service recovery based on health status
  3. Resource Limits: Memory and CPU constraints to prevent resource exhaustion
  4. Monitoring Scripts: Custom scripts for comprehensive health analysis
  5. Alerting: Critical service notifications and error detection

Configuration Files#

  • docker-compose.health.yml - Health checks and restart policies overlay
  • scripts/health-monitor.sh - Comprehensive monitoring script
  • docker-compose.dev.yml - Base configuration with health checks

Health Check Configuration#

Application Services#

All application services implement standardized health checks:

yaml
healthcheck:
  test: ['CMD', 'curl', '-f', 'http://localhost:8080/health']
  interval: 30s # Check every 30 seconds
  timeout: 10s # Timeout after 10 seconds
  retries: 3 # Allow 3 failures before marking unhealthy
  start_period: 60s # Grace period for startup

Restart Policies#

Services use different restart strategies based on criticality:

yaml
restart: unless-stopped # Restart unless manually stopped

Resource Limits#

Memory constraints prevent resource exhaustion:

yaml
deploy:
  resources:
    limits:
      memory: 512M # Maximum memory
    reservations:
      memory: 256M # Guaranteed memory

Service Categories#

Core Services (Critical)#

  • bff: Backend for Frontend
  • auth: Authentication Service
  • conversation: Conversation Management
  • ai: AI/LLM Service

Memory Allocation: 512MB limit, 256MB reserved

Resource-Intensive Services#

  • tts: Text-to-Speech (2GB limit, 1GB reserved)
  • stt: Speech-to-Text (2GB limit, 1GB reserved)
  • ai: AI Service with models (1GB limit, 512MB reserved)
  • rag: RAG/Vector Service (1GB limit, 512MB reserved)

Database Services#

  • postgres: PostgreSQL Database (2GB limit, 1GB reserved)
  • elasticsearch: Search Engine (2GB limit, 1GB reserved)
  • neo4j: Graph Database (2GB limit, 1GB reserved)
  • milvus: Vector Database (4GB limit, 2GB reserved)

Monitoring Scripts#

health-monitor.sh#

Comprehensive monitoring script with multiple modes:

bash
# Check all services
./scripts/health-monitor.sh all

# Watch continuously (every 30 seconds)
./scripts/health-monitor.sh watch

# Watch with custom interval
./scripts/health-monitor.sh watch 60

# Critical service alerts
./scripts/health-monitor.sh alerts

# Generate health report
./scripts/health-monitor.sh report

# Check specific service categories
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases
./scripts/health-monitor.sh utilities

Features#

  1. Service Health Status: Real-time health checks for all services
  2. Response Time Monitoring: Latency measurements for service endpoints
  3. Resource Usage: CPU and memory consumption tracking
  4. Error Detection: Automatic error log analysis
  5. Container Status: Docker container state monitoring
  6. System Health: Overall platform status summary

Health Endpoints#

Standard Health Response#

All services implement a standardized health endpoint:

javascript
// GET /health
{
  "status": "healthy",
  "timestamp": "2025-01-06T12:00:00Z",
  "service": "service-name",
  "version": "1.0.0",
  "uptime": 3600,
  "dependencies": {
    "database": "healthy",
    "redis": "healthy"
  },
  "checks": {
    "memory": "ok",
    "disk": "ok",
    "external_apis": "ok"
  }
}

Service-Specific Endpoints#

  • AI Service: /v1/health - Provider status and model availability
  • Database Services: Database-specific health checks
  • Utilities: Service-appropriate health endpoints

Restart Strategies#

Automatic Restart#

Services restart automatically when:

  • Health check fails 3 consecutive times
  • Container crashes or exits unexpectedly
  • Resource limits exceeded

Manual Recovery#

bash
# Restart specific service
docker compose -f docker-compose.dev.yml restart bff

# Restart all services
docker compose -f docker-compose.dev.yml restart

# Force recreation
docker compose -f docker-compose.dev.yml up -d --force-recreate

Dependency Management#

Services with dependencies wait for upstream services:

yaml
depends_on:
  postgres:
    condition: service_healthy
  redis:
    condition: service_healthy

Monitoring and Alerting#

Critical Service Monitoring#

Core services are monitored for immediate alerts:

  • BFF (Backend for Frontend)
  • Authentication Service
  • Conversation Service
  • AI Service

Alert Conditions#

  • Service health check failures
  • High error rates in logs
  • Resource exhaustion
  • Dependency failures
  • Response time degradation

Log Monitoring#

Automatic detection of error patterns:

  • Error keywords: "error", "exception", "failed"
  • Stack traces
  • HTTP 5xx responses
  • Database connection failures

Performance Monitoring#

Response Time Tracking#

Health checks measure response times:

  • Normal: < 1000ms
  • Warning: 1000-5000ms
  • Critical: > 5000ms

Resource Usage Monitoring#

  • CPU Usage: Container CPU utilization
  • Memory Usage: Container memory consumption
  • Disk Usage: Available disk space
  • Network I/O: Network traffic patterns

Maintenance and Troubleshooting#

Common Issues#

  1. Services Not Starting

    bash
    # Check health status
    ./scripts/health-monitor.sh all
    
    # View logs
    docker logs lilith-bff-1
    
  2. High Memory Usage

    bash
    # Check resource usage
    ./scripts/health-monitor.sh docker
    
    # Restart memory-intensive services
    docker compose restart ai rag tts stt
    
  3. Database Connection Issues

    bash
    # Check database health
    ./scripts/health-monitor.sh databases
    
    # Verify database connectivity
    docker exec lilith-postgres-1 pg_isready -U lilith -d lilith_dev
    

Maintenance Tasks#

  1. Health Check Tuning

    • Adjust intervals for different service types
    • Customize timeout values based on service characteristics
    • Optimize start periods for slower services
  2. Resource Limit Optimization

    • Monitor actual resource usage patterns
    • Adjust memory limits based on requirements
    • Scale resources for production workloads
  3. Alert Configuration

    • Configure notification channels
    • Set up alert escalation policies
    • Customize alert thresholds

Production Considerations#

High Availability#

  • Service Redundancy: Multiple instances for critical services
  • Load Balancing: Health check-based routing
  • Graceful Degradation: Fallback mechanisms for service failures

Monitoring at Scale#

  • Metrics Collection: Prometheus/Grafana integration
  • Log Aggregation: Centralized logging with ELK stack
  • Distributed Tracing: Jaeger for request tracing
  • Alert Management: PagerDuty/Slack integrations

Disaster Recovery#

  • Health Check Backup: Secondary monitoring systems
  • Automated Failover: Health-based traffic routing
  • Data Backup: Automated database backups
  • Recovery Procedures: Documented recovery playbooks

Usage Examples#

Development Environment#

bash
# Start with health monitoring
docker compose -f docker-compose.dev.yml -f docker-compose.health.yml up -d

# Monitor health
./scripts/health-monitor.sh watch

# Check specific services
./scripts/health-monitor.sh core
./scripts/health-monitor.sh databases

Test Environment#

bash
# Start test environment with health checks
docker compose -f docker-compose.dev.yml -f docker-compose.test.yml -f docker-compose.health.yml up -d

# Run health checks with alerts
./scripts/health-monitor.sh alerts

Production Monitoring#

bash
# Continuous monitoring
nohup ./scripts/health-monitor.sh watch 60 > health.log 2>&1 &

# Generate daily reports
./scripts/health-monitor.sh report > daily-health-$(date +%Y%m%d).txt

Best Practices#

  1. Regular Health Checks: Monitor services continuously
  2. Proactive Maintenance: Address issues before they become critical
  3. Resource Planning: Monitor resource usage trends
  4. Documentation: Keep health configurations updated
  5. Testing: Regularly test recovery procedures
  6. Alert Tuning: Balance sensitivity and false positives
  7. Performance Monitoring: Track response time trends
  8. Capacity Planning: Scale resources based on monitoring data