# Rollback Procedures This document describes rollback procedures for the Oshun monorepo, covering ECS deployments, database migrations, infrastructure changes, and the historical Lilith/Yemaya migration. --- ## Quick Reference | Scenario | Action | Section | | ------------------------- | ----------------------------- | --------- | | **ECS Deployment** | Redeploy previous task def | Section 1 | | **Database Migration** | Rollback via Prisma/Knex | Section 2 | | **Service Code Rollback** | Revert specific commits | Section 3 | | **Infrastructure** | Docker/Terraform rollback | Section 4 | | **Historical Migration** | Restore Lilith/Yemaya bundles | Section 5 | --- ## 1. ECS Deployment Rollback ### Revert to Previous Task Definition ```bash # List recent task definitions for a service aws ecs list-task-definitions --family-prefix oshun-tara-api --sort DESC --max-items 5 # Update service to use previous task definition aws ecs update-service \ --cluster oshun-production \ --service tara-api \ --task-definition oshun-tara-api: # Monitor the deployment aws ecs describe-services --cluster oshun-production --services tara-api \ --query 'services[0].deployments' ``` ### Force New Deployment (Same Version) ```bash aws ecs update-service \ --cluster oshun-production \ --service tara-api \ --force-new-deployment ``` ### Check Service Health ```bash # View service events aws ecs describe-services --cluster oshun-production --services tara-api \ --query 'services[0].events[:5]' # View running tasks aws ecs list-tasks --cluster oshun-production --service-name tara-api ``` --- ## 2. Database Rollback ### Prisma Migration Rollback Most domains use Prisma for database migrations: ```bash cd libs/{domain}/database # List migrations npx prisma migrate status # Mark a migration as rolled back npx prisma migrate resolve --rolled-back # Full database reset (WARNING: destroys data) npx prisma migrate reset ``` ### Knex Migration Rollback (Lilith) Lilith uses Knex for migrations: ```bash cd libs/lilith/database # Rollback last migration npx knex migrate:rollback # Rollback all migrations npx knex migrate:rollback --all ``` ### PostgreSQL Backup/Restore For critical rollbacks, restore from database backups: ```bash # Create a backup before risky operations pg_dump -h localhost -U oshun -d tara > tara-backup-$(date +%Y%m%d).sql # Restore from backup psql -h localhost -U oshun -d tara < tara-backup-20260214.sql ``` --- ## 3. Service Code Rollback ### Revert Specific Commits ```bash # Find the commits to revert git log --oneline -20 # Revert a specific commit git revert # Revert a range of commits git revert ^.. ``` ### Restore Specific Files ```bash # Restore a specific file from a known-good commit git checkout -- path/to/file # Restore an entire directory git checkout -- apps/tara/api/ ``` --- ## 4. Infrastructure Rollback ### Docker Compose (Development) ```bash # Stop all containers docker compose -f docker/docker-compose.dev.yml down # Remove volumes if needed (WARNING: destroys data) docker compose -f docker/docker-compose.dev.yml down -v # Restart fresh docker compose -f docker/docker-compose.dev.yml up -d ``` ### Terraform ```bash # Review what would change cd infra/terraform/tara-api terraform plan # Rollback to previous state (if state is saved) terraform apply -target= -var-file= ``` --- ## 5. Historical: Lilith/Yemaya Migration Rollback > **Note**: This section documents rollback procedures from the original > Lilith/Yemaya repository consolidation into the Oshun monorepo (January 2026). > These bundles are kept for historical reference and emergency recovery. ### Backup Locations | Backup | Location | Size | Created | | ------------- | ----------------------------------------------------------------- | ---- | ---------- | | Lilith Bundle | `/home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle` | 47MB | 2026-01-10 | | Yemaya Bundle | `/home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle` | 13MB | 2026-01-10 | ### Restore from Bundles ```bash # Verify bundle integrity git bundle verify /home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle git bundle verify /home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle # Clone from bundle into a separate directory mkdir -p /home/ubuntu/oshun-restored git clone /home/ubuntu/oshun/backups/lilith-pre-migration-20260110.bundle /home/ubuntu/oshun-restored/lilith git clone /home/ubuntu/oshun/backups/yemaya-pre-migration-20260110.bundle /home/ubuntu/oshun-restored/yemaya ``` --- ## 6. Verification Checklist After any rollback, verify the following: ### Code Verification - [ ] `git status` shows clean working directory - [ ] `git log -1` shows expected commit - [ ] All expected files are present ### Build Verification ```bash pnpm install pnpm typecheck pnpm lint pnpm test ``` ### Service Verification - [ ] All services start without errors - [ ] Health checks pass (`/health` endpoint returns 200) - [ ] API endpoints respond correctly - [ ] Database connections work - [ ] Redis connections work ### Data Verification - [ ] Database schemas match expected state - [ ] No data loss occurred - [ ] Foreign key constraints are valid --- ## 7. Rollback Decision Matrix | Symptom | Severity | Action | | -------------------- | -------- | -------------------------------------- | | Build fails | Medium | Revert commits (Section 3) | | Tests fail | Medium | Service code rollback (Section 3) | | Services don't start | High | ECS rollback (Section 1) + investigate | | Database errors | Critical | Database rollback (Section 2) | | Data corruption | Critical | DB restore from backup + code rollback | | Production outage | Critical | ECS rollback + infrastructure rollback | --- ## 8. Post-Rollback Actions After completing a rollback: 1. **Document the Issue** - What failed? - When did it fail? - What was the error message? - What rollback procedure was used? 2. **Create an Incident Report** - Timeline of events - Root cause analysis - Impact assessment - Prevention measures 3. **Communicate** - Notify stakeholders - Update status page if applicable - Send team notification --- ## Revision History | Version | Date | Author | Changes | | ------- | ---------- | ---------------- | ------------------------------------------------- | | 1.0 | 2026-01-10 | Development Team | Initial version (migration-focused) | | 1.1 | 2026-02-14 | Development Team | Generalized for ongoing use; added ECS procedures |