Purpose: Provide fast, safe rollback steps for production deployments using canary guardrails and Helm history.
When to Trigger#
- Automated guardrails fail in CI (
auto-rollbackjob) or health checks degrade during canary. - Error budgets breached (5xx spikes, p95 latency > 1s) within 10 minutes of rollout.
- Critical regression confirmed by smoke tests or on-call triage.
Automated Path (Preferred)#
- CI auto-rollback fires when post-deploy sampling fails.
- Pipeline triggers
helm rollback <service>-productionto the previous revision. - Incident commander verifies health:bash
kubectl rollout status deploy/<service>-production -n production --timeout=300s curl -f https://<service>.lilith.app/health - Post in
#api-alertswith impact, cause, and rollback timestamp.
Manual Rollback (CLI)#
Use when automation is insufficient or partial rollback is required.
bash
SERVICE=bff
NAMESPACE=production
# Check history
helm history "$SERVICE-production" -n "$NAMESPACE"
# Roll back to previous revision
helm rollback "$SERVICE-production" 1 -n "$NAMESPACE"
# Monitor rollout
kubectl rollout status deploy/$SERVICE-production -n "$NAMESPACE" --timeout=600s
# Verify health + latency
for i in {1..5}; do curl -f https://$SERVICE.lilith.app/health && sleep 5; done
Traffic Cutover After Canary Failure#
- Keep canary pods scaled but remove from ingress:bash
kubectl scale deploy/$SERVICE-canary -n production --replicas=0 - Patch ingress to route 100% to stable:bash
kubectl patch ingress $SERVICE -n production --type='json' \ -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"'"$SERVICE"'-stable"}]'
Data / Schema Rollbacks#
- If migrations are involved, validate backward compatibility before rollback.
- Use pre-deployment backups (
s3://db-backups/<service>/) for destructive changes. - For forward-only migrations, gate rollout with feature flags and shadow reads.
Local Development Rollback (Docker Compose)#
For local development environments using Docker Compose:
Quick Rollback#
bash
# Stop all services and remove containers
docker compose -f docker-compose.dev.yml down
# Rebuild and restart (clean slate)
docker compose -f docker-compose.dev.yml build --no-cache
docker compose -f docker-compose.dev.yml up -d
Rollback to Previous Container State#
bash
# View container history
docker ps -a --filter "name=lilith" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.CreatedAt}}"
# Stop current containers
docker compose -f docker-compose.dev.yml stop
# Restart with previous image versions (if cached)
docker compose -f docker-compose.dev.yml up -d --no-build
Database State Rollback#
bash
# Backup current database state (PostgreSQL)
docker exec lilith-postgres-1 pg_dump -U lilith lilith_dev > backup_$(date +%Y%m%d_%H%M%S).sql
# Restore from backup
cat backup_YYYYMMDD_HHMMSS.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev
# Reset database to initial state
docker compose -f docker-compose.dev.yml stop postgres
docker volume rm lilith_postgres_data
docker compose -f docker-compose.dev.yml up -d postgres
cat dev/init-db.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev
Service-Specific Rollback#
bash
# Rollback single service
SERVICE=bff
docker compose -f docker-compose.dev.yml stop $SERVICE
docker compose -f docker-compose.dev.yml rm -f $SERVICE
docker compose -f docker-compose.dev.yml build $SERVICE --no-cache
docker compose -f docker-compose.dev.yml up -d $SERVICE
After Action#
- Capture timeline, detection source, and fix in the incident record.
- Add regression test covering the root cause.
- Validate dashboards/alerts caught the issue; close any gaps.
- Update this runbook if new steps were required.