Eve library

Rollback Plan

Tracker

Revision 1

On this page

Historical task source. The Eve task board holds current task state.

This document outlines procedures for rolling back Tara services in case of critical issues after deployment.


Table of Contents#

  1. Rollback Decision Framework
  2. API Rollback
  3. Web Application Rollback
  4. Mobile Application Rollback
  5. Database Rollback
  6. Communication Plan
  7. Post-Rollback Procedures

Rollback Decision Framework#

When to Rollback#

A rollback should be considered when:

Severity Criteria Decision
P0 Service completely unavailable Immediate rollback
P0 Data loss or corruption Immediate rollback
P0 Security breach Immediate rollback
P1 Core feature broken (auth, playback) Rollback within 1 hour
P1 Payment processing failures Rollback within 1 hour
P2 Significant UX degradation Evaluate vs hotfix
P2 Performance degradation > 50% Evaluate vs hotfix
P3 Minor feature broken Hotfix preferred

Decision Tree#

text
Issue Detected
    │
    ├── Is the service down or data at risk?
    │   ├── YES → Immediate rollback
    │   └── NO ↓
    │
    ├── Is a core feature broken?
    │   ├── YES → Can it be hotfixed in < 1 hour?
    │   │   ├── YES → Attempt hotfix
    │   │   └── NO → Rollback
    │   └── NO ↓
    │
    ├── Is the issue affecting > 50% of users?
    │   ├── YES → Rollback or feature flag disable
    │   └── NO → Investigate and plan fix
    │
    └── END

Authorization#

Rollback Type Who Can Authorize
API/Backend On-call engineer, Engineering Lead
Web App On-call engineer, Engineering Lead
Database Engineering Lead, Database Admin
Mobile (halt) Engineering Lead, Product Owner
Mobile (full) Executive + Engineering Lead

API Rollback#

Prerequisites#

  • Access to deployment platform (AWS ECS / Vercel)
  • AWS CLI configured with appropriate permissions
  • Previous stable image tag identified

Vercel Deployment Rollback#

For API deployed on Vercel:

bash
# List recent deployments
vercel list tara-api --token $VERCEL_TOKEN

# Identify the last stable deployment URL
# Format: tara-api-xxxxxxxxx-team.vercel.app

# Rollback to specific deployment
vercel rollback [DEPLOYMENT_URL] --token $VERCEL_TOKEN

# Verify rollback
curl -I https://api.tara.app/health

AWS ECS Rollback#

For API deployed on AWS ECS:

bash
# 1. Identify current and previous task definitions
aws ecs describe-services \
  --cluster tara-production \
  --services tara-api \
  --query 'services[0].taskDefinition'

# 2. List recent task definitions
aws ecs list-task-definitions \
  --family-prefix tara-api \
  --sort DESC \
  --max-items 5

# 3. Update service to previous task definition
aws ecs update-service \
  --cluster tara-production \
  --service tara-api \
  --task-definition tara-api:PREVIOUS_REVISION \
  --force-new-deployment

# 4. Monitor deployment
aws ecs describe-services \
  --cluster tara-production \
  --services tara-api \
  --query 'services[0].deployments'

# 5. Verify health
curl -I https://api.tara.app/health

API Rollback Checklist#

  • Notify team in #incidents channel
  • Identify stable version to rollback to
  • Execute rollback command
  • Verify health endpoint returns 200
  • Test critical endpoints (auth, playback)
  • Monitor error rates for 15 minutes
  • Update status page
  • Document in incident log

Estimated Time#

  • Vercel: 1-2 minutes
  • AWS ECS: 3-5 minutes

Web Application Rollback#

Vercel Rollback#

bash
# List recent deployments
vercel list tara-web --token $VERCEL_TOKEN

# Rollback to previous deployment
vercel rollback [DEPLOYMENT_URL] --token $VERCEL_TOKEN

# Verify
curl -I https://tara.app

Manual Rollback (if automated fails)#

bash
# 1. Clone repository if not available
git clone git@github.com:GreyChimp/oshun.git
cd oshun

# 2. Checkout previous stable tag
git checkout tags/web-v1.0.0  # Replace with actual tag

# 3. Deploy manually
cd apps/tara-web
vercel --prod --token $VERCEL_TOKEN

Feature Flag Rollback#

For issues isolated to specific features:

typescript
// Disable feature without full rollback
// Update feature flags in config or database

// Example: Disable new meditation player
await updateFeatureFlag('new_meditation_player', false);

// Users will fall back to previous implementation

Web Rollback Checklist#

  • Notify team in #incidents channel
  • Identify stable deployment
  • Execute rollback
  • Clear CDN cache if needed
  • Verify home page loads
  • Test login flow
  • Test meditation playback
  • Monitor for 15 minutes

Estimated Time#

  • Vercel: 1-2 minutes
  • CDN cache clear: 1-5 minutes

Mobile Application Rollback#

Mobile rollbacks are more complex due to app store review processes.

Immediate Mitigation (Server-Side)#

Before attempting store rollback, try server-side mitigations:

typescript
// 1. Feature flags
// Disable problematic feature for affected app versions
if (appVersion >= '1.2.0' && appVersion < '1.2.1') {
  features.newFeature = false;
}

// 2. Force update prompt
// Require users to update to a working version
if (appVersion === '1.2.0') {
  return { forceUpdate: true, minVersion: '1.1.9' };
}

// 3. API version routing
// Route old app versions to stable API
// Configure at API gateway level

iOS App Store Rollback#

Option 1: Halt Phased Release (If Using)#

text
1. Open App Store Connect
2. Go to App > iOS App > App Store
3. Select the problematic version
4. Click "Halt Phased Release"
5. Users with auto-update on will not receive the update

Option 2: Submit Previous Version#

text
1. Identify last stable version (e.g., 1.1.9)
2. Update version number to 1.2.1 (must be higher)
3. Build with previous stable code
4. Submit for expedited review
5. In submission notes, explain the urgency
6. Wait for approval (expedited: 24-48 hours)

Option 3: Emergency Contact Apple#

For critical security issues:

text
1. Contact Apple Developer Support
2. Explain the security vulnerability
3. Request emergency app removal if necessary
4. Submit fixed version for expedited review

Google Play Rollback#

Option 1: Halt Staged Rollout#

text
1. Open Google Play Console
2. Go to Release > Production
3. Find the problematic release
4. Click "Halt rollout"
5. Staged rollout stops at current percentage

Option 2: Rollback to Previous Version#

text
1. Open Google Play Console
2. Go to Release > Production
3. Click "Create new release"
4. Upload APK/AAB from previous stable version
5. Update version code (must be higher)
6. Submit for review
7. Review time: Usually same day

Option 3: Full Rollback#

text
1. Open Google Play Console
2. Go to Release > Production
3. Find problematic release
4. Click "Manage" > "Rollback"
5. Confirm rollback to previous version
6. Users will receive previous version on next update

Mobile Rollback Checklist#

  • Notify team in #incidents channel
  • Implement server-side mitigation first
  • Determine appropriate rollback strategy
  • For iOS: Halt release or submit new version
  • For Android: Halt rollout or rollback
  • Monitor crash reports
  • Communicate with affected users
  • Update status page

Estimated Time#

Action iOS Android
Server-side mitigation Immediate Immediate
Halt release/rollout Immediate Immediate
Submit new version 24-48 hours 2-24 hours
Full removal 24-48 hours Same day

Database Rollback#

Database rollbacks are the most sensitive. Proceed with extreme caution.

Prerequisites#

  • Access to database admin tools
  • Point-in-time recovery enabled
  • Recent backup verified
  • Maintenance window (if possible)

Schema Rollback (Migrations)#

For reversible migration issues:

bash
# 1. Identify current migration version
npm run db:status

# 2. Rollback to previous migration
npm run db:rollback -- --step 1

# 3. Verify schema state
npm run db:status

# 4. Test affected functionality

Point-in-Time Recovery (PITR)#

For data corruption or accidental deletion:

bash
# AWS RDS Example

# 1. Identify target time (before issue occurred)
TARGET_TIME="2026-01-23T10:30:00Z"

# 2. Create new instance from point-in-time
aws rds restore-db-instance-to-point-in-time \
  --source-db-instance-identifier tara-production \
  --target-db-instance-identifier tara-production-restored \
  --restore-time $TARGET_TIME

# 3. Wait for restoration (can take 30-60 minutes)
aws rds wait db-instance-available \
  --db-instance-identifier tara-production-restored

# 4. Update application connection strings to new instance
# OR
# 5. Rename instances to swap
aws rds modify-db-instance \
  --db-instance-identifier tara-production \
  --new-db-instance-identifier tara-production-old

aws rds modify-db-instance \
  --db-instance-identifier tara-production-restored \
  --new-db-instance-identifier tara-production

Data-Only Rollback#

For specific data issues without schema changes:

sql
-- 1. Identify affected data
SELECT * FROM users WHERE updated_at > '2026-01-23 10:30:00';

-- 2. Restore from backup table (if available)
INSERT INTO users
SELECT * FROM users_backup
WHERE id IN (SELECT id FROM users WHERE updated_at > '2026-01-23 10:30:00')
ON CONFLICT (id) DO UPDATE SET
  column1 = EXCLUDED.column1,
  column2 = EXCLUDED.column2;

-- 3. Or restore from audit log
-- Application-specific procedure

Database Rollback Checklist#

  • STOP all write operations if possible
  • Notify all teams immediately
  • Assess scope of data impact
  • Choose rollback strategy:
    • Migration rollback (schema changes only)
    • Point-in-time recovery (full restoration)
    • Selective data restoration
  • Execute rollback with verification at each step
  • Test application functionality
  • Verify data integrity
  • Resume write operations
  • Document data loss (if any)
  • Notify affected users

Estimated Time#

Method Time
Migration rollback 1-5 minutes
Point-in-time recovery 30-60 minutes
Selective data restore Variable

Communication Plan#

Internal Communication#

Immediate (Within 5 Minutes)#

text
Channel: #incidents (Slack/Teams)

@channel INCIDENT: [Service] rollback initiated

Impact: [Brief description]
Status: Rollback in progress
Lead: [Name]

Updates to follow.

During Rollback#

text
Channel: #incidents

UPDATE: [Service] rollback

- Rollback command executed
- Waiting for deployment/recovery
- Estimated completion: X minutes

Current status: [In progress/Completed/Failed]

After Rollback#

text
Channel: #incidents

RESOLVED: [Service] rollback complete

- Service restored at [time]
- Root cause: [Brief description]
- Next steps: [Investigation/Postmortem scheduled]

Incident log: [Link]

External Communication#

Status Page Update#

text
Title: Service Degradation - [Service Name]

Body:
We are currently experiencing issues with [affected feature].
Our team is actively working to resolve this.

Impact: [What users may experience]
Workaround: [If any]

Last updated: [Timestamp]

User Notification (If Major Impact)#

text
Subject: Service Update - Tara

Hi [Name],

We recently experienced a technical issue that may have affected
your experience with Tara. The issue has been resolved.

What happened: [Brief, non-technical explanation]
What we did: [Brief resolution summary]
Your data: [Assurance about data safety]

We apologize for any inconvenience. If you have any questions,
please contact support@tara.app.

Thank you for your patience,
The Tara Team

Communication Templates#

Situation Internal Channel External Channel
Rollback started #incidents Status page
Rollback progress #incidents Status page (major)
Rollback complete #incidents Status page
Data impact #incidents, exec Email to affected
Extended outage #incidents Social media, email

Post-Rollback Procedures#

Immediate (First Hour)#

  1. Verify Stability

    • Monitor error rates for 30 minutes
    • Check all critical user flows
    • Verify no cascading issues
  2. Document Timeline

    text
    [Time] Issue detected
    [Time] Rollback decision made
    [Time] Rollback executed
    [Time] Service restored
    [Time] Stability confirmed
    
  3. Gather Artifacts

    • Error logs from the incident
    • Metrics during the incident
    • User reports/complaints
    • Deployment logs

Short-Term (First 24 Hours)#

  1. Root Cause Analysis

    • Why did the deployment fail?
    • Why wasn't it caught in testing?
    • What was the blast radius?
  2. Fix Development

    • Develop proper fix for the issue
    • Add regression tests
    • Review with team
  3. Stakeholder Update

    • Brief summary to leadership
    • Customer impact assessment
    • Timeline for proper fix

Long-Term (First Week)#

  1. Postmortem Meeting

    • Blameless review of incident
    • Timeline walkthrough
    • Contributing factors
    • Action items
  2. Process Improvements

    • Update testing procedures
    • Improve monitoring/alerting
    • Update rollback documentation
    • Training if needed
  3. Documentation

    • Update runbooks
    • Document lessons learned
    • Share across team

Postmortem Template#

markdown
# Incident Postmortem: [Title]

**Date:** [Date] **Duration:** [Start time] - [End time] **Impact:**
[Description of user impact] **Severity:** P0/P1/P2

## Summary

[2-3 sentence summary of what happened]

## Timeline

- [Time] Event
- [Time] Event
- [Time] Event

## Root Cause

[Technical explanation of why this happened]

## Contributing Factors

- [Factor 1]
- [Factor 2]

## Resolution

[How the issue was resolved]

## Impact

- Users affected: [Number]
- Revenue impact: [If applicable]
- Data impact: [If applicable]

## Lessons Learned

### What went well

- [Item]

### What could be improved

- [Item]

## Action Items

| Action   | Owner  | Due Date | Status   |
| -------- | ------ | -------- | -------- |
| [Action] | [Name] | [Date]   | [Status] |

## Attendees

[List of postmortem meeting attendees]

Quick Reference Card#

Emergency Contacts#

Role Contact
On-Call oncall@tara.app
Engineering engineering@tara.app
Database Admin dba@tara.app
Security security@tara.app

Rollback Commands#

bash
# Vercel (API/Web)
vercel rollback [URL] --token $VERCEL_TOKEN

# AWS ECS
aws ecs update-service --cluster tara-production \
  --service tara-api --task-definition tara-api:PREVIOUS

# Database Migration
npm run db:rollback -- --step 1

# Feature Flag Disable
curl -X POST https://api.tara.app/admin/flags \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"flag": "feature_name", "enabled": false}'

Verification Endpoints#

bash
# API Health
curl https://api.tara.app/health

# Web Health
curl https://tara.app

# Database Connection
curl https://api.tara.app/health/db

Last Updated: January 2026

For questions about rollback procedures, contact ops@tara.app