Eve library

Deployment Rollback Runbook

Runbook

Revision 1

On this page

Purpose: Provide fast, safe rollback steps for production deployments using canary guardrails and Helm history.

When to Trigger#

  • Automated guardrails fail in CI (auto-rollback job) or health checks degrade during canary.
  • Error budgets breached (5xx spikes, p95 latency > 1s) within 10 minutes of rollout.
  • Critical regression confirmed by smoke tests or on-call triage.

Automated Path (Preferred)#

  1. CI auto-rollback fires when post-deploy sampling fails.
  2. Pipeline triggers helm rollback <service>-production to the previous revision.
  3. Incident commander verifies health:
    bash
    kubectl rollout status deploy/<service>-production -n production --timeout=300s
    curl -f https://<service>.lilith.app/health
    
  4. Post in #api-alerts with impact, cause, and rollback timestamp.

Manual Rollback (CLI)#

Use when automation is insufficient or partial rollback is required.

bash
SERVICE=bff
NAMESPACE=production

# Check history
helm history "$SERVICE-production" -n "$NAMESPACE"

# Roll back to previous revision
helm rollback "$SERVICE-production" 1 -n "$NAMESPACE"

# Monitor rollout
kubectl rollout status deploy/$SERVICE-production -n "$NAMESPACE" --timeout=600s

# Verify health + latency
for i in {1..5}; do curl -f https://$SERVICE.lilith.app/health && sleep 5; done

Traffic Cutover After Canary Failure#

  • Keep canary pods scaled but remove from ingress:
    bash
    kubectl scale deploy/$SERVICE-canary -n production --replicas=0
    
  • Patch ingress to route 100% to stable:
    bash
    kubectl patch ingress $SERVICE -n production --type='json' \
      -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"'"$SERVICE"'-stable"}]'
    

Data / Schema Rollbacks#

  • If migrations are involved, validate backward compatibility before rollback.
  • Use pre-deployment backups (s3://db-backups/<service>/) for destructive changes.
  • For forward-only migrations, gate rollout with feature flags and shadow reads.

Local Development Rollback (Docker Compose)#

For local development environments using Docker Compose:

Quick Rollback#

bash
# Stop all services and remove containers
docker compose -f docker-compose.dev.yml down

# Rebuild and restart (clean slate)
docker compose -f docker-compose.dev.yml build --no-cache
docker compose -f docker-compose.dev.yml up -d

Rollback to Previous Container State#

bash
# View container history
docker ps -a --filter "name=lilith" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.CreatedAt}}"

# Stop current containers
docker compose -f docker-compose.dev.yml stop

# Restart with previous image versions (if cached)
docker compose -f docker-compose.dev.yml up -d --no-build

Database State Rollback#

bash
# Backup current database state (PostgreSQL)
docker exec lilith-postgres-1 pg_dump -U lilith lilith_dev > backup_$(date +%Y%m%d_%H%M%S).sql

# Restore from backup
cat backup_YYYYMMDD_HHMMSS.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev

# Reset database to initial state
docker compose -f docker-compose.dev.yml stop postgres
docker volume rm lilith_postgres_data
docker compose -f docker-compose.dev.yml up -d postgres
cat dev/init-db.sql | docker exec -i lilith-postgres-1 psql -U lilith lilith_dev

Service-Specific Rollback#

bash
# Rollback single service
SERVICE=bff
docker compose -f docker-compose.dev.yml stop $SERVICE
docker compose -f docker-compose.dev.yml rm -f $SERVICE
docker compose -f docker-compose.dev.yml build $SERVICE --no-cache
docker compose -f docker-compose.dev.yml up -d $SERVICE

After Action#

  • Capture timeline, detection source, and fix in the incident record.
  • Add regression test covering the root cause.
  • Validate dashboards/alerts caught the issue; close any gaps.
  • Update this runbook if new steps were required.