Overview#
This document describes the comprehensive environment configuration management system for the Lilith platform, supporting multiple deployment environments (development, staging, production) with environment-specific configurations, security policies, and deployment automation.
Architecture#
Environment Types#
- Development (dev): Local development with debugging features enabled
- Staging: Pre-production testing environment
- Production (prod): Live production environment with full security
Directory Structure#
text
config/
├── environments/
│ ├── development/
│ │ ├── .env # Environment variables
│ │ ├── services/ # Service-specific configs
│ │ │ ├── bff/
│ │ │ ├── ai/
│ │ │ └── ...
│ │ ├── vault/ # Vault configurations
│ │ ├── monitoring/ # Monitoring configs
│ │ ├── networking/ # Network policies
│ │ └── security/ # Security policies
│ ├── staging/
│ │ └── [same structure as development]
│ └── production/
│ └── [same structure as development]
├── active -> environments/development # Symlink to active environment
└── .env.active # Current environment file
Configuration Components#
1. Environment Variables (.env)#
Each environment has its own .env file with:
Development Environment#
bash
NODE_ENV=development
ENVIRONMENT=development
DEBUG=true
HOT_RELOAD=true
LOG_LEVEL=debug
# URLs
API_BASE_URL=http://localhost:8080
WEB_BASE_URL=http://localhost:3000
VAULT_ADDR=http://localhost:8200
# Database
DB_HOST=localhost
DB_PORT=5433
DB_NAME=lilith_dev
DB_USER=lilith
DB_PASSWORD=dev_password
# Resources (generous)
CPU_LIMIT=2
MEMORY_LIMIT=4Gi
Production Environment#
bash
NODE_ENV=production
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=warn
# URLs
API_BASE_URL=https://api.lilith.app
WEB_BASE_URL=https://lilith.app
VAULT_ADDR=https://vault.lilith.app
# Database
DB_HOST=postgres-prod.internal
DB_PORT=5432
DB_NAME=lilith_prod
DB_USER=lilith_prod
DB_PASSWORD=
# Security
ENCRYPTION_ENABLED=true
AUDIT_LOGGING_ENABLED=true
RATE_LIMITING_ENABLED=true
# Resources (optimized)
CPU_LIMIT=1
MEMORY_LIMIT=1Gi
2. Docker Compose Overrides#
Each environment gets its own Docker Compose override file:
Development (docker-compose.development.yml)#
yaml
services:
bff:
environment:
- NODE_ENV=development
- DEBUG=true
volumes:
- ./services/bff/src:/app/src
- ./config/environments/development/.env:/app/.env:ro
command: ['npm', 'run', 'dev']
# Development tools
nginx-proxy:
image: nginx:alpine
ports:
- '8081:80'
Production (docker-compose.production.yml)#
yaml
services:
bff:
environment:
- NODE_ENV=production
- LOG_LEVEL=warn
restart: unless-stopped
deploy:
replicas: 2
resources:
limits:
memory: 1Gi
reservations:
memory: 512Mi
update_config:
parallelism: 1
delay: 10s
failure_action: rollback
3. Vault Configuration#
Environment-specific Vault configurations:
Development Vault (config/environments/development/vault/vault.hcl)#
hcl
ui = true
api_addr = "http://0.0.0.0:8200"
cluster_addr = "http://0.0.0.0:8201"
backend "file" {
path = "/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
cluster_address = "0.0.0.0:8201"
}
log_level = "debug"
Production Vault (config/environments/production/vault/vault.hcl)#
hcl
ui = true
api_addr = "https://vault.lilith.app"
cluster_addr = "https://vault.lilith.app:8201"
backend "file" {
path = "/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = false
cluster_address = "0.0.0.0:8201"
}
log_level = "info"
4. Service Configurations#
Each service gets environment-specific configuration:
Service Configuration JSON (config/environments/production/services/bff/config.json)#
json
{
"service": "bff",
"environment": "production",
"version": "1.0.0",
"features": {
"healthChecks": true,
"metrics": true,
"tracing": true,
"debug": false
},
"endpoints": {
"health": "/health",
"metrics": "/metrics",
"readiness": "/ready"
},
"resources": {
"cpu": {
"request": "100m",
"limit": "500m"
},
"memory": {
"request": "256Mi",
"limit": "1Gi"
}
},
"environmentSpecific": {
"debug": false,
"logLevel": "warn",
"performanceMonitoring": true,
"errorReporting": true
}
}
Usage#
1. Environment Setup#
Generate All Configurations#
bash
# Generate configurations for all environments
./scripts/config-manager.sh all
# Generate for specific environment
./scripts/config-manager.sh generate development
./scripts/config-manager.sh generate staging
./scripts/config-manager.sh generate production
Validate Configuration#
bash
# Validate specific environment
./scripts/config-manager.sh validate production
# This checks:
# • Required files exist
# • JSON configurations are valid
# • Docker compose files are valid
# • No missing service configurations
2. Environment Switching#
Quick Switch#
bash
# Switch to production environment
./scripts/config-manager.sh switch production
# Switch to development
./scripts/config-manager.sh switch dev
Using Environment Switch Script#
bash
# Switch environments
./scripts/switch-env.sh prod
./scripts/switch-env.sh staging
./scripts/switch-env.sh dev
# Show current environment
./scripts/switch-env.sh show
3. Deployment#
Development Deployment#
bash
# Switch to development environment
./scripts/switch-env.sh dev
# Deploy development services
./scripts/deploy/development/deploy.sh
Staging Deployment#
bash
# Switch to staging environment
./scripts/switch-env.sh staging
# Deploy staging services
./scripts/deploy/staging/deploy.sh
Production Deployment#
bash
# Switch to production environment
./scripts/switch-env.sh production
# Deploy production services
./scripts/deploy/production/deploy.sh
4. Health Monitoring#
Environment-Specific Health Checks#
bash
# Check current environment
./scripts/switch-env.sh show
# Run health check for current environment
./scripts/deploy/$(echo $ENVIRONMENT | tr '[:upper:]' '[:lower:]')/health-check.sh
Configuration Management Workflow#
1. Development Workflow#
-
Local Setup
bash./scripts/config-manager.sh generate development ./scripts/switch-env.sh dev -
Development Services
bashdocker compose -f docker-compose.yml -f docker-compose.development.yml up -d -
Feature Development
- Modify service code
- Test locally
- Validate configuration changes
2. Testing Workflow#
-
Staging Setup
bash./scripts/config-manager.sh generate staging ./scripts/switch-env.sh staging -
Deploy to Staging
bash./scripts/deploy/staging/deploy.sh -
Integration Testing
- Run automated tests
- Validate service integration
- Performance testing
3. Production Workflow#
-
Production Setup
bash./scripts/config-manager.sh generate production ./scripts/switch-env.sh production -
Pre-deployment Validation
bash./scripts/config-manager.sh validate production -
Production Deployment
bash./scripts/deploy/production/deploy.sh -
Post-deployment Verification
bash./scripts/deploy/production/health-check.sh
Security Considerations#
1. Configuration Security#
Secrets Management#
- Never commit secrets to version control
- Use Vault for all sensitive data
- Environment-specific secret paths in Vault
- Automatic rotation for critical secrets
Configuration Validation#
- JSON schema validation for service configs
- Docker compose validation before deployment
- Environment variable validation for required settings
2. Network Security#
Development Environment#
- Localhost access only
- Debug endpoints enabled
- Mock services for external APIs
Staging Environment#
- Internal network access only
- Real services (with staging data)
- Security features enabled
Production Environment#
- Full security features enabled
- Load balancer with SSL termination
- Network policies and service mesh
- External access through controlled endpoints
3. Access Control#
Environment Access#
bash
# Development: All developers
# Staging: Development and QA teams
# Production: DevOps and authorized personnel only
Service Configuration#
bash
# Production services:
# - No debug endpoints
# - Limited log levels
# - Error reporting only
# - Performance monitoring enabled
Monitoring and Observability#
1. Environment Monitoring#
Service Health#
bash
# Health check endpoints for each service
GET /health
GET /ready
GET /metrics
Configuration Monitoring#
bash
# Track configuration changes
# Validate environment consistency
# Monitor service restarts
2. Alerting#
Environment-Specific Alerts#
- Development: Error notifications
- Staging: Performance and error alerts
- Production: Critical alerts with escalation
Configuration Drift Detection#
bash
# Compare configurations
./scripts/config-manager.sh compare development staging
./scripts/config-manager.sh compare staging production
Best Practices#
1. Configuration Management#
Version Control#
- Configuration files in version control
- Secrets never committed
- Environment templates for consistency
Change Management#
- Review process for production changes
- Staging validation before production
- Rollback plans for critical changes
2. Environment Consistency#
Standardization#
- Consistent structure across environments
- Standard naming conventions
- Required fields validation
Testing#
- Configuration validation in CI/CD
- Environment testing before deployment
- Rollback testing procedures
3. Operational Excellence#
Automation#
- Automated deployment pipelines
- Configuration validation scripts
- Health monitoring and alerting
Documentation#
- Comprehensive documentation for each environment
- Runbooks for common operations
- Troubleshooting guides for issues
Troubleshooting#
1. Common Issues#
Configuration Validation Failures#
bash
# Check configuration syntax
./scripts/config-manager.sh validate production
# Check JSON files
jq empty config/environments/production/services/*/config.json
Environment Switch Failures#
bash
# Check directory permissions
ls -la config/environments/
ls -la config/environments/active
# Check symlink
ls -la config/.env.active
Deployment Failures#
bash
# Check environment variables
cat config/environments/active/.env
# Check docker compose files
docker compose -f docker-compose.yml -f docker-compose.production.yml config
2. Debug Configuration#
Enable Debug Mode#
bash
# Add debug logging
export DEBUG_CONFIG=true
export LOG_LEVEL=debug
# Check configuration loading
./scripts/deploy/$(echo $ENVIRONMENT | tr '[:upper:]' '[:lower:]')/deploy.sh --dry-run
Validate Service Configuration#
bash
# Check service config files
cat config/environments/$(echo $ENVIRONMENT | tr '[:upper:]' '[:lower:]')/services/bff/config.json
# Validate JSON
jq empty config/environments/$(echo $ENVIRONMENT | tr '[:upper:]' '[:lower:]')/services/bff/config.json
3. Health Monitoring#
Service Health#
bash
# Check service health
curl -f http://localhost:8080/health
# Check all services
./scripts/health-monitor.sh all
Configuration Health#
bash
# Check current environment
./scripts/switch-env.sh show
# Validate configuration
./scripts/config-manager.sh validate $ENVIRONMENT
Advanced Features#
1. Configuration Templates#
Template Generation#
bash
# Generate new environment template
./scripts/config-manager.sh generate custom-environment
# Customize based on existing environment
cp -r config/environments/production config/environments/custom-environment
# Modify as needed
2. Environment-Specific Features#
Feature Flags#
json
{
"features": {
"experimentalFeatures": {
"development": true,
"staging": false,
"production": false
},
"advancedMonitoring": {
"development": false,
"staging": true,
"production": true
}
}
}
Resource Scaling#
json
{
"resources": {
"autoScaling": {
"development": false,
"staging": true,
"production": true
},
"maxReplicas": {
"development": 1,
"staging": 2,
"production": 5
}
}
}
3. Multi-Region Support#
Region Configuration#
bash
# Add region-specific configurations
mkdir -p config/environments/production/us-east-1
mkdir -p config/environments/production/us-west-2
# Region-specific overrides
echo "REGION=us-east-1" >> config/environments/production/us-east-1/.env
echo "CLUSTER=lilith-prod-use1" >> config/environments/production/us-east-1/.env
Integration with CI/CD#
1. GitHub Actions#
Environment Configuration#
yaml
name: Deploy to Production
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Generate Production Config
run: ./scripts/config-manager.sh generate production
- name: Validate Configuration
run: ./scripts/config-manager.sh validate production
- name: Deploy to Production
run: ./scripts/deploy/production/deploy.sh
2. Environment Variables in CI/CD#
Secure Configuration Loading#
bash
# Load environment-specific configuration
source config/environments/$ENVIRONMENT/.env
# Use configuration in deployment
docker compose -f docker-compose.yml -f docker-compose.$ENVIRONMENT.yml up -d
3. Testing Pipeline#
Multi-Environment Testing#
yaml
# Staging tests
- name: Run Staging Tests
run: |
./scripts/switch-env.sh staging
./scripts/deploy/staging/deploy.sh
./scripts/deploy/staging/health-check.sh
npm run test:e2e:staging
# Production validation
- name: Production Validation
run: |
./scripts/config-manager.sh validate production
./scripts/deploy/production/deploy.sh --dry-run
Migration Guide#
1. Existing Environment Migration#
Backup Current Configuration#
bash
# Create backup directory
mkdir -p backups/config-migration-$(date +%Y%m%d)
# Backup current configurations
cp -r docker-compose.* backups/config-migration-$(date +%Y%m%d)/
cp -r .env* backups/config-migration-$(date +%Y%m%d)/ 2>/dev/null || true
Migrate to New System#
bash
# Generate new configurations
./scripts/config-manager.sh all
# Validate new configurations
for env in development staging production; do
./scripts/config-manager.sh validate $env
done
# Test migration
./scripts/switch-env.sh development
# Test services work with new configuration
2. Service Migration#
Update Service Configuration Loading#
javascript
// In service startup
import config from './config.json';
// Load environment-specific overrides
const env = process.env.ENVIRONMENT || 'development';
const envConfigPath = `config/environments/${env}/services/${process.env.SERVICE_NAME}/config.json`;
try {
const envConfig = require(envConfigPath);
config = { ...config, ...envConfig };
} catch (error) {
console.warn(`No environment-specific config found, using defaults`);
}
Configuration Validation#
javascript
// Validate required configuration
const requiredFields = ['service', 'environment', 'endpoints'];
for (const field of requiredFields) {
if (!config[field]) {
throw new Error(`Missing required configuration: ${field}`);
}
}
References#
- Configuration Scripts:
scripts/config-manager.sh - Environment Switch:
scripts/switch-env.sh - Deployment Scripts:
scripts/deploy/*/ - Health Monitoring:
scripts/health-monitor.sh - Vault Setup:
docs/deploy/VAULT_SETUP.md - Secret Rotation:
docs/deploy/VAULT_MIGRATION.md