Purpose: Safe deployment practices for Lilith microservices using GitOps and rolling updates
Deployment Architecture Overview#
GitOps Workflow#
Developer → Git Push → CI Pipeline → Image Build → GitOps Repo Update → ArgoCD → Kubernetes Deployment
↓ ↓ ↓ ↓ ↓ ↓ ↓
Feature Trigger Tests Container Manifest Sync Rolling
Branch Build Pass Registry Update Deploy Update
Deployment Environments#
Development → Staging → Production
↓ ↓ ↓
Feature Integration Production
Testing Testing Workloads
↓ ↓ ↓
Immediate Automated Manual
Deploy Deploy Approval
Pre-Deployment Checklist#
Code Review Requirements#
- Code review approved by at least 2 engineers
- All tests passing (unit, integration, e2e)
- Security scan completed without high/critical issues
- Documentation updated for API changes
- Database migrations tested in staging
Infrastructure Readiness#
- Target environment healthy (all services green)
- Resource capacity sufficient for new deployment
- Database migrations completed if required
- External dependencies verified
- Monitoring and alerting configured
Deployment Planning#
- Deployment window scheduled (avoid peak hours)
- Rollback plan documented and tested
- Feature flags configured for gradual rollout
- Communication plan for user-facing changes
- On-call engineer available during deployment
Deployment Types and Procedures#
Standard Rolling Deployment#
For Stateless Services (BFF, Auth, Conversation)#
# 1. Verify current deployment health
kubectl get deployments -n lilith-bff
kubectl rollout status deployment bff-deployment -n lilith-bff
# 2. Update image tag in ArgoCD application
argocd app set lilith-bff --parameter image.tag=v2.1.3
# 3. Sync deployment
argocd app sync lilith-bff
# 4. Monitor rollout progress
kubectl rollout status deployment bff-deployment -n lilith-bff --timeout=600s
# 5. Verify health after deployment
curl -f https://bff.lilith.com/v1/health
./scripts/post-deploy-validation.sh bff
Rolling Update Configuration#
apiVersion: apps/v1
kind: Deployment
metadata:
name: bff-deployment
namespace: lilith-bff
spec:
replicas: 6
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1 # Only 1 pod down at a time
maxSurge: 2 # Up to 2 extra pods during update
template:
spec:
containers:
- name: bff
image: lilith/bff:v2.1.3
readinessProbe:
httpGet:
path: /v1/health
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
livenessProbe:
httpGet:
path: /v1/health
port: 8080
initialDelaySeconds: 30
periodSeconds: 30
Blue-Green Deployment#
For Critical Services Requiring Zero Downtime#
# 1. Deploy to green environment (inactive)
kubectl apply -f manifests/bff-green-deployment.yaml
# 2. Wait for green deployment to be ready
kubectl wait --for=condition=Available deployment bff-green-deployment -n lilith-bff --timeout=600s
# 3. Run validation tests against green environment
./scripts/validate-green-deployment.sh bff
# 4. Switch traffic to green (update service selector)
kubectl patch service bff-service -n lilith-bff \
--patch='{"spec":{"selector":{"version":"green"}}}'
# 5. Monitor for any issues
sleep 300
./scripts/monitor-deployment-health.sh bff 300
# 6. If successful, scale down blue deployment
kubectl scale deployment bff-blue-deployment --replicas=0 -n lilith-bff
# 7. If issues detected, rollback immediately
# kubectl patch service bff-service -n lilith-bff \
# --patch='{"spec":{"selector":{"version":"blue"}}}'
Canary Deployment#
For Gradual Feature Rollouts#
# 1. Deploy canary version (5% traffic)
kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: bff-rollout
namespace: lilith-bff
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 5
- pause: {duration: 10m}
- setWeight: 20
- pause: {duration: 10m}
- setWeight: 50
- pause: {duration: 10m}
- setWeight: 100
canaryService: bff-canary-service
stableService: bff-stable-service
trafficRouting:
istio:
virtualService:
name: bff-vs
routes:
- primary
selector:
matchLabels:
app: bff
template:
spec:
containers:
- name: bff
image: lilith/bff:v2.1.3-canary
EOF
# 2. Monitor canary metrics
kubectl argo rollouts get rollout bff-rollout -n lilith-bff --watch
# 3. Promote if metrics look good, or abort if issues
# kubectl argo rollouts promote bff-rollout -n lilith-bff
# kubectl argo rollouts abort bff-rollout -n lilith-bff
Database Migrations#
Safe Migration Procedures#
Pre-Migration Validation#
# 1. Backup database before migration
kubectl exec -it postgres-primary-0 -n database -- \
pg_dump lilith > /backups/pre-migration-$(date +%Y%m%d-%H%M).sql
# 2. Test migration on staging database
kubectl exec -it postgres-staging-0 -n staging -- \
psql lilith < migrations/v2.1.3-migration.sql
# 3. Validate migration results
kubectl exec -it postgres-staging-0 -n staging -- \
psql lilith -c "SELECT count(*) FROM new_table;"
# 4. Test application compatibility with new schema
./scripts/test-app-with-new-schema.sh staging
Production Migration Execution#
# 1. Put application in maintenance mode (if required)
kubectl scale deployment --replicas=0 -n lilith-bff
kubectl apply -f manifests/maintenance-page.yaml
# 2. Run migration with monitoring
kubectl exec -it postgres-primary-0 -n database -- \
psql lilith < migrations/v2.1.3-migration.sql | tee migration.log
# 3. Validate migration success
kubectl exec -it postgres-primary-0 -n database -- \
psql lilith -c "\d+ new_table;"
# 4. Deploy application with new schema support
kubectl scale deployment bff-deployment --replicas=6 -n lilith-bff
kubectl wait --for=condition=Available deployment bff-deployment -n lilith-bff --timeout=600s
# 5. Remove maintenance mode
kubectl delete -f manifests/maintenance-page.yaml
Backward-Compatible Migration Pattern#
-- Phase 1: Add new column (nullable)
ALTER TABLE users ADD COLUMN new_feature_flag BOOLEAN DEFAULT NULL;
-- Deploy application version that can handle both old and new schema
-- Phase 2: Backfill data (in next deployment)
UPDATE users SET new_feature_flag = false WHERE new_feature_flag IS NULL;
-- Phase 3: Make column NOT NULL (in subsequent deployment)
ALTER TABLE users ALTER COLUMN new_feature_flag SET NOT NULL;
ALTER TABLE users ALTER COLUMN new_feature_flag SET DEFAULT false;
Rollback Procedures#
Automated Rollback Triggers#
#!/bin/bash
# Automated rollback script based on health metrics
SERVICE=$1
NAMESPACE="lilith-${SERVICE}"
ROLLBACK_THRESHOLD_ERROR_RATE=5.0
ROLLBACK_THRESHOLD_RESPONSE_TIME=5000
# Get current error rate and response time
error_rate=$(kubectl exec -n monitoring prometheus-0 -- \
promtool query instant "rate(http_requests_total{job=\"$SERVICE\",status=~\"5..\"}[5m]) * 100")
response_time=$(kubectl exec -n monitoring prometheus-0 -- \
promtool query instant "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"$SERVICE\"}[5m])) * 1000")
# Check if rollback is needed
if (( $(echo "$error_rate > $ROLLBACK_THRESHOLD_ERROR_RATE" | bc -l) )) || \
(( $(echo "$response_time > $ROLLBACK_THRESHOLD_RESPONSE_TIME" | bc -l) )); then
echo "Rollback triggered for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"
# Execute rollback
kubectl rollout undo deployment ${SERVICE}-deployment -n $NAMESPACE
# Wait for rollback to complete
kubectl rollout status deployment ${SERVICE}-deployment -n $NAMESPACE --timeout=300s
# Notify team
./scripts/send-alert.sh "🔄 Automatic rollback executed for $SERVICE due to performance degradation"
# Create incident for investigation
./scripts/create-incident.sh "Automatic rollback: $SERVICE performance degradation"
else
echo "No rollback needed for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"
fi
Manual Rollback Process#
# 1. Identify current and previous versions
kubectl rollout history deployment bff-deployment -n lilith-bff
# 2. Rollback to previous version
kubectl rollout undo deployment bff-deployment -n lilith-bff
# 3. Or rollback to specific revision
kubectl rollout undo deployment bff-deployment --to-revision=3 -n lilith-bff
# 4. Monitor rollback progress
kubectl rollout status deployment bff-deployment -n lilith-bff
# 5. Verify service health after rollback
curl -f https://bff.lilith.com/v1/health
./scripts/post-rollback-validation.sh bff
# 6. Update ArgoCD to match rolled-back state
argocd app set lilith-bff --parameter image.tag=v2.1.2
Feature Flag Management#
Feature Flag Deployment Pattern#
# 1. Deploy new feature with flag disabled
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: feature-flags
namespace: lilith-bff
data:
NEW_CHAT_INTERFACE: "false"
ADVANCED_SEARCH: "false"
VOICE_CHAT_V2: "false"
EOF
# 2. Deploy application code that checks feature flags
kubectl set image deployment/bff-deployment bff=lilith/bff:v2.1.3-with-flags -n lilith-bff
# 3. Gradually enable feature for user segments
kubectl patch configmap feature-flags -n lilith-bff \
--patch='{"data":{"NEW_CHAT_INTERFACE":"true:beta-users"}}'
# 4. Monitor feature performance
./scripts/monitor-feature-flags.sh NEW_CHAT_INTERFACE
# 5. Full rollout if metrics are good
kubectl patch configmap feature-flags -n lilith-bff \
--patch='{"data":{"NEW_CHAT_INTERFACE":"true"}}'
# 6. Emergency disable if issues
kubectl patch configmap feature-flags -n lilith-bff \
--patch='{"data":{"NEW_CHAT_INTERFACE":"false"}}'
Deployment Monitoring and Validation#
Post-Deployment Validation Script#
#!/bin/bash
# Comprehensive post-deployment validation
SERVICE=$1
NAMESPACE="lilith-${SERVICE}"
echo "Starting post-deployment validation for $SERVICE..."
# 1. Check pod health
echo "Checking pod health..."
kubectl get pods -n $NAMESPACE -l app=$SERVICE
if [ $(kubectl get pods -n $NAMESPACE -l app=$SERVICE --field-selector=status.phase=Running --no-headers | wc -l) -eq 0 ]; then
echo "❌ No running pods found for $SERVICE"
exit 1
fi
# 2. Check service endpoints
echo "Checking service endpoints..."
kubectl get endpoints -n $NAMESPACE
if [ $(kubectl get endpoints $SERVICE-service -n $NAMESPACE -o jsonpath='{.subsets[0].addresses}' | jq length) -eq 0 ]; then
echo "❌ No service endpoints available for $SERVICE"
exit 1
fi
# 3. Health check endpoint
echo "Testing health endpoint..."
if ! curl -f https://$SERVICE.lilith.com/v1/health > /dev/null 2>&1; then
echo "❌ Health endpoint check failed for $SERVICE"
exit 1
fi
# 4. Functional tests
echo "Running functional tests..."
if ! ./tests/functional-tests-$SERVICE.sh; then
echo "❌ Functional tests failed for $SERVICE"
exit 1
fi
# 5. Performance baseline check
echo "Checking performance metrics..."
response_time=$(curl -w "%{time_total}" -s -o /dev/null https://$SERVICE.lilith.com/v1/health)
if (( $(echo "$response_time > 2.0" | bc -l) )); then
echo "⚠️ Response time higher than expected: ${response_time}s"
fi
# 6. Check for new errors
echo "Checking for deployment-related errors..."
error_count=$(kubectl logs -n $NAMESPACE -l app=$SERVICE --since=10m | grep -i error | wc -l)
if [ $error_count -gt 10 ]; then
echo "⚠️ High error count detected: $error_count errors in last 10 minutes"
fi
echo "✅ Post-deployment validation completed successfully for $SERVICE"
Deployment Metrics Dashboard#
#!/bin/bash
# Create deployment metrics dashboard
create_deployment_dashboard() {
local service=$1
curl -X POST http://grafana.lilith.com/api/dashboards/db \
-H "Content-Type: application/json" \
-d '{
"dashboard": {
"title": "'$service' Deployment Metrics",
"panels": [
{
"title": "Response Time",
"type": "graph",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"'$service'\"}[5m]))"
}
]
},
{
"title": "Error Rate",
"type": "graph",
"targets": [
{
"expr": "rate(http_requests_total{job=\"'$service'\",status=~\"5..\"}[5m]) * 100"
}
]
},
{
"title": "Pod Restarts",
"type": "graph",
"targets": [
{
"expr": "rate(kube_pod_container_status_restarts_total{pod=~\"'$service'-.*\"}[5m])"
}
]
}
]
}
}'
}
# Create dashboards for all services
for service in bff auth conversation content media ai; do
create_deployment_dashboard $service
done
Deployment Automation#
CI/CD Pipeline Integration#
# .github/workflows/deploy.yml
name: Deploy to Production
on:
push:
tags:
- 'v*'
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Run security scan
run: |
trivy fs --severity HIGH,CRITICAL .
- name: Build and push image
run: |
docker build -t lilith/${{ github.event.repository.name }}:${{ github.ref_name }} .
docker push lilith/${{ github.event.repository.name }}:${{ github.ref_name }}
- name: Update GitOps repo
run: |
git clone https://github.com/lilith/gitops-manifests.git
cd gitops-manifests
yq eval '.spec.template.spec.containers[0].image = "lilith/${{ github.event.repository.name }}:${{ github.ref_name }}"' \
-i ${{ github.event.repository.name }}/deployment.yaml
git commit -am "Deploy ${{ github.event.repository.name }}:${{ github.ref_name }}"
git push
- name: Wait for ArgoCD sync
run: |
argocd app wait lilith-${{ github.event.repository.name }} --timeout 600
- name: Run post-deployment tests
run: |
./scripts/post-deploy-validation.sh ${{ github.event.repository.name }}
- name: Notify team
if: failure()
run: |
./scripts/send-alert.sh "❌ Deployment failed for ${{ github.event.repository.name }}:${{ github.ref_name }}"
Deployment Troubleshooting Guide#
Common Issues and Solutions#
1. Container Fails to Start#
Symptoms:
- Pod in
CrashLoopBackOfforErrorstate - Service health check fails immediately
Diagnosis:
# Check pod events
kubectl describe pod <pod-name> -n <namespace>
# View container logs
kubectl logs <pod-name> -n <namespace> --previous
# Check for resource constraints
kubectl top pods -n <namespace>
Common Causes and Fixes: | Cause | Solution | |-------|----------| | Missing environment variables | Check ConfigMaps and Secrets are mounted | | Port already in use | Verify port conflicts in deployment spec | | Insufficient memory/CPU | Increase resource limits in deployment | | Failed health probe | Check probe paths and initial delays |
2. Database Connection Failures#
Symptoms:
- Service starts but returns 500 errors
- Logs show connection refused or timeout
Diagnosis:
# Check database service is running
kubectl get svc -n lilith-databases
# Test connectivity from pod
kubectl exec -it <pod> -- nc -z postgres 5432
# Check connection pool status
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep db_pool
Fixes:
# Restart database if needed (CAUTION: may cause brief downtime)
kubectl rollout restart statefulset postgres -n lilith-databases
# Check database credentials
kubectl get secret db-credentials -n <namespace> -o yaml
# Verify network policies allow traffic
kubectl get networkpolicy -n <namespace>
3. Service-to-Service Communication Failures#
Symptoms:
- Intermittent 503 errors
- Timeouts on internal service calls
Diagnosis:
# Check service discovery
kubectl get endpoints <service-name> -n <namespace>
# Test internal DNS resolution
kubectl exec -it <pod> -- nslookup <service>.<namespace>.svc.cluster.local
# Check network policies
kubectl describe networkpolicy -n <namespace>
Fixes:
- Verify service selectors match pod labels
- Check if circuit breaker is open (see logs for circuit breaker state)
- Increase timeout values if services are under load
4. Memory/CPU Issues#
Symptoms:
- Pod OOMKilled
- Slow response times
- Throttling warnings in metrics
Diagnosis:
# Check resource usage
kubectl top pods -n <namespace>
# Check limits vs requests
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Limits\|Requests"
# Check for memory leaks
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep nodejs_heap
Fixes:
# Increase resources in deployment
resources:
requests:
memory: '512Mi'
cpu: '250m'
limits:
memory: '1Gi'
cpu: '500m'
5. Docker Compose (Local Development) Issues#
Symptoms:
- Services fail to start locally
- Port conflicts
- Volume mounting issues
Common Fixes:
# Clean up and restart
docker compose -f docker-compose.dev.yml down -v
docker system prune -f
docker compose -f docker-compose.dev.yml up -d --build
# Port conflict - find what's using the port
lsof -i :3000
# Volume permissions
sudo chown -R $(id -u):$(id -g) ./volumes
# Clear npm cache if node_modules issues
rm -rf node_modules
npm ci
6. Database Migrations Fail#
Symptoms:
- Service starts but throws migration errors
- Schema mismatch errors
Diagnosis:
# Check migration status
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev -c "SELECT * FROM schema_migrations;"
# View migration logs
kubectl logs <pod-name> -n <namespace> | grep migration
Fixes:
# Rollback last migration (if safe)
npm run migrate:rollback
# Reset migrations (CAUTION: data loss in dev only)
npm run migrate:reset
# Manual fix in database
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev
Emergency Contacts#
| Role | Contact | When to Use |
|---|---|---|
| On-Call Engineer | PagerDuty | Production issues |
| Database Admin | #database-team | Data integrity issues |
| Security Team | #security-oncall | Security incidents |
| Platform Lead | @platform-lead | Architecture decisions |
Quick Reference Commands#
# Full stack restart (local)
npm run deploy:restart
# View all service logs
npm run deploy:logs
# Health check all services
npm run health:check
# Force rebuild single service
npm run build:service -- <service-name>
Remember: Deployments should be boring and predictable. Always have a rollback plan, monitor actively during deployments, and validate thoroughly before considering a deployment complete.