Historical task source. The Eve task board holds current task state.
This document outlines procedures for rolling back Tara services in case of critical issues after deployment.
Table of Contents#
- Rollback Decision Framework
- API Rollback
- Web Application Rollback
- Mobile Application Rollback
- Database Rollback
- Communication Plan
- Post-Rollback Procedures
Rollback Decision Framework#
When to Rollback#
A rollback should be considered when:
| Severity | Criteria | Decision |
|---|---|---|
| P0 | Service completely unavailable | Immediate rollback |
| P0 | Data loss or corruption | Immediate rollback |
| P0 | Security breach | Immediate rollback |
| P1 | Core feature broken (auth, playback) | Rollback within 1 hour |
| P1 | Payment processing failures | Rollback within 1 hour |
| P2 | Significant UX degradation | Evaluate vs hotfix |
| P2 | Performance degradation > 50% | Evaluate vs hotfix |
| P3 | Minor feature broken | Hotfix preferred |
Decision Tree#
Issue Detected
│
├── Is the service down or data at risk?
│ ├── YES → Immediate rollback
│ └── NO ↓
│
├── Is a core feature broken?
│ ├── YES → Can it be hotfixed in < 1 hour?
│ │ ├── YES → Attempt hotfix
│ │ └── NO → Rollback
│ └── NO ↓
│
├── Is the issue affecting > 50% of users?
│ ├── YES → Rollback or feature flag disable
│ └── NO → Investigate and plan fix
│
└── END
Authorization#
| Rollback Type | Who Can Authorize |
|---|---|
| API/Backend | On-call engineer, Engineering Lead |
| Web App | On-call engineer, Engineering Lead |
| Database | Engineering Lead, Database Admin |
| Mobile (halt) | Engineering Lead, Product Owner |
| Mobile (full) | Executive + Engineering Lead |
API Rollback#
Prerequisites#
- Access to deployment platform (AWS ECS / Vercel)
- AWS CLI configured with appropriate permissions
- Previous stable image tag identified
Vercel Deployment Rollback#
For API deployed on Vercel:
# List recent deployments
vercel list tara-api --token $VERCEL_TOKEN
# Identify the last stable deployment URL
# Format: tara-api-xxxxxxxxx-team.vercel.app
# Rollback to specific deployment
vercel rollback [DEPLOYMENT_URL] --token $VERCEL_TOKEN
# Verify rollback
curl -I https://api.tara.app/health
AWS ECS Rollback#
For API deployed on AWS ECS:
# 1. Identify current and previous task definitions
aws ecs describe-services \
--cluster tara-production \
--services tara-api \
--query 'services[0].taskDefinition'
# 2. List recent task definitions
aws ecs list-task-definitions \
--family-prefix tara-api \
--sort DESC \
--max-items 5
# 3. Update service to previous task definition
aws ecs update-service \
--cluster tara-production \
--service tara-api \
--task-definition tara-api:PREVIOUS_REVISION \
--force-new-deployment
# 4. Monitor deployment
aws ecs describe-services \
--cluster tara-production \
--services tara-api \
--query 'services[0].deployments'
# 5. Verify health
curl -I https://api.tara.app/health
API Rollback Checklist#
- Notify team in #incidents channel
- Identify stable version to rollback to
- Execute rollback command
- Verify health endpoint returns 200
- Test critical endpoints (auth, playback)
- Monitor error rates for 15 minutes
- Update status page
- Document in incident log
Estimated Time#
- Vercel: 1-2 minutes
- AWS ECS: 3-5 minutes
Web Application Rollback#
Vercel Rollback#
# List recent deployments
vercel list tara-web --token $VERCEL_TOKEN
# Rollback to previous deployment
vercel rollback [DEPLOYMENT_URL] --token $VERCEL_TOKEN
# Verify
curl -I https://tara.app
Manual Rollback (if automated fails)#
# 1. Clone repository if not available
git clone git@github.com:GreyChimp/oshun.git
cd oshun
# 2. Checkout previous stable tag
git checkout tags/web-v1.0.0 # Replace with actual tag
# 3. Deploy manually
cd apps/tara-web
vercel --prod --token $VERCEL_TOKEN
Feature Flag Rollback#
For issues isolated to specific features:
// Disable feature without full rollback
// Update feature flags in config or database
// Example: Disable new meditation player
await updateFeatureFlag('new_meditation_player', false);
// Users will fall back to previous implementation
Web Rollback Checklist#
- Notify team in #incidents channel
- Identify stable deployment
- Execute rollback
- Clear CDN cache if needed
- Verify home page loads
- Test login flow
- Test meditation playback
- Monitor for 15 minutes
Estimated Time#
- Vercel: 1-2 minutes
- CDN cache clear: 1-5 minutes
Mobile Application Rollback#
Mobile rollbacks are more complex due to app store review processes.
Immediate Mitigation (Server-Side)#
Before attempting store rollback, try server-side mitigations:
// 1. Feature flags
// Disable problematic feature for affected app versions
if (appVersion >= '1.2.0' && appVersion < '1.2.1') {
features.newFeature = false;
}
// 2. Force update prompt
// Require users to update to a working version
if (appVersion === '1.2.0') {
return { forceUpdate: true, minVersion: '1.1.9' };
}
// 3. API version routing
// Route old app versions to stable API
// Configure at API gateway level
iOS App Store Rollback#
Option 1: Halt Phased Release (If Using)#
1. Open App Store Connect
2. Go to App > iOS App > App Store
3. Select the problematic version
4. Click "Halt Phased Release"
5. Users with auto-update on will not receive the update
Option 2: Submit Previous Version#
1. Identify last stable version (e.g., 1.1.9)
2. Update version number to 1.2.1 (must be higher)
3. Build with previous stable code
4. Submit for expedited review
5. In submission notes, explain the urgency
6. Wait for approval (expedited: 24-48 hours)
Option 3: Emergency Contact Apple#
For critical security issues:
1. Contact Apple Developer Support
2. Explain the security vulnerability
3. Request emergency app removal if necessary
4. Submit fixed version for expedited review
Google Play Rollback#
Option 1: Halt Staged Rollout#
1. Open Google Play Console
2. Go to Release > Production
3. Find the problematic release
4. Click "Halt rollout"
5. Staged rollout stops at current percentage
Option 2: Rollback to Previous Version#
1. Open Google Play Console
2. Go to Release > Production
3. Click "Create new release"
4. Upload APK/AAB from previous stable version
5. Update version code (must be higher)
6. Submit for review
7. Review time: Usually same day
Option 3: Full Rollback#
1. Open Google Play Console
2. Go to Release > Production
3. Find problematic release
4. Click "Manage" > "Rollback"
5. Confirm rollback to previous version
6. Users will receive previous version on next update
Mobile Rollback Checklist#
- Notify team in #incidents channel
- Implement server-side mitigation first
- Determine appropriate rollback strategy
- For iOS: Halt release or submit new version
- For Android: Halt rollout or rollback
- Monitor crash reports
- Communicate with affected users
- Update status page
Estimated Time#
| Action | iOS | Android |
|---|---|---|
| Server-side mitigation | Immediate | Immediate |
| Halt release/rollout | Immediate | Immediate |
| Submit new version | 24-48 hours | 2-24 hours |
| Full removal | 24-48 hours | Same day |
Database Rollback#
Database rollbacks are the most sensitive. Proceed with extreme caution.
Prerequisites#
- Access to database admin tools
- Point-in-time recovery enabled
- Recent backup verified
- Maintenance window (if possible)
Schema Rollback (Migrations)#
For reversible migration issues:
# 1. Identify current migration version
npm run db:status
# 2. Rollback to previous migration
npm run db:rollback -- --step 1
# 3. Verify schema state
npm run db:status
# 4. Test affected functionality
Point-in-Time Recovery (PITR)#
For data corruption or accidental deletion:
# AWS RDS Example
# 1. Identify target time (before issue occurred)
TARGET_TIME="2026-01-23T10:30:00Z"
# 2. Create new instance from point-in-time
aws rds restore-db-instance-to-point-in-time \
--source-db-instance-identifier tara-production \
--target-db-instance-identifier tara-production-restored \
--restore-time $TARGET_TIME
# 3. Wait for restoration (can take 30-60 minutes)
aws rds wait db-instance-available \
--db-instance-identifier tara-production-restored
# 4. Update application connection strings to new instance
# OR
# 5. Rename instances to swap
aws rds modify-db-instance \
--db-instance-identifier tara-production \
--new-db-instance-identifier tara-production-old
aws rds modify-db-instance \
--db-instance-identifier tara-production-restored \
--new-db-instance-identifier tara-production
Data-Only Rollback#
For specific data issues without schema changes:
-- 1. Identify affected data
SELECT * FROM users WHERE updated_at > '2026-01-23 10:30:00';
-- 2. Restore from backup table (if available)
INSERT INTO users
SELECT * FROM users_backup
WHERE id IN (SELECT id FROM users WHERE updated_at > '2026-01-23 10:30:00')
ON CONFLICT (id) DO UPDATE SET
column1 = EXCLUDED.column1,
column2 = EXCLUDED.column2;
-- 3. Or restore from audit log
-- Application-specific procedure
Database Rollback Checklist#
- STOP all write operations if possible
- Notify all teams immediately
- Assess scope of data impact
- Choose rollback strategy:
- Migration rollback (schema changes only)
- Point-in-time recovery (full restoration)
- Selective data restoration
- Execute rollback with verification at each step
- Test application functionality
- Verify data integrity
- Resume write operations
- Document data loss (if any)
- Notify affected users
Estimated Time#
| Method | Time |
|---|---|
| Migration rollback | 1-5 minutes |
| Point-in-time recovery | 30-60 minutes |
| Selective data restore | Variable |
Communication Plan#
Internal Communication#
Immediate (Within 5 Minutes)#
Channel: #incidents (Slack/Teams)
@channel INCIDENT: [Service] rollback initiated
Impact: [Brief description]
Status: Rollback in progress
Lead: [Name]
Updates to follow.
During Rollback#
Channel: #incidents
UPDATE: [Service] rollback
- Rollback command executed
- Waiting for deployment/recovery
- Estimated completion: X minutes
Current status: [In progress/Completed/Failed]
After Rollback#
Channel: #incidents
RESOLVED: [Service] rollback complete
- Service restored at [time]
- Root cause: [Brief description]
- Next steps: [Investigation/Postmortem scheduled]
Incident log: [Link]
External Communication#
Status Page Update#
Title: Service Degradation - [Service Name]
Body:
We are currently experiencing issues with [affected feature].
Our team is actively working to resolve this.
Impact: [What users may experience]
Workaround: [If any]
Last updated: [Timestamp]
User Notification (If Major Impact)#
Subject: Service Update - Tara
Hi [Name],
We recently experienced a technical issue that may have affected
your experience with Tara. The issue has been resolved.
What happened: [Brief, non-technical explanation]
What we did: [Brief resolution summary]
Your data: [Assurance about data safety]
We apologize for any inconvenience. If you have any questions,
please contact support@tara.app.
Thank you for your patience,
The Tara Team
Communication Templates#
| Situation | Internal Channel | External Channel |
|---|---|---|
| Rollback started | #incidents | Status page |
| Rollback progress | #incidents | Status page (major) |
| Rollback complete | #incidents | Status page |
| Data impact | #incidents, exec | Email to affected |
| Extended outage | #incidents | Social media, email |
Post-Rollback Procedures#
Immediate (First Hour)#
-
Verify Stability
- Monitor error rates for 30 minutes
- Check all critical user flows
- Verify no cascading issues
-
Document Timeline
text[Time] Issue detected [Time] Rollback decision made [Time] Rollback executed [Time] Service restored [Time] Stability confirmed -
Gather Artifacts
- Error logs from the incident
- Metrics during the incident
- User reports/complaints
- Deployment logs
Short-Term (First 24 Hours)#
-
Root Cause Analysis
- Why did the deployment fail?
- Why wasn't it caught in testing?
- What was the blast radius?
-
Fix Development
- Develop proper fix for the issue
- Add regression tests
- Review with team
-
Stakeholder Update
- Brief summary to leadership
- Customer impact assessment
- Timeline for proper fix
Long-Term (First Week)#
-
Postmortem Meeting
- Blameless review of incident
- Timeline walkthrough
- Contributing factors
- Action items
-
Process Improvements
- Update testing procedures
- Improve monitoring/alerting
- Update rollback documentation
- Training if needed
-
Documentation
- Update runbooks
- Document lessons learned
- Share across team
Postmortem Template#
# Incident Postmortem: [Title]
**Date:** [Date] **Duration:** [Start time] - [End time] **Impact:**
[Description of user impact] **Severity:** P0/P1/P2
## Summary
[2-3 sentence summary of what happened]
## Timeline
- [Time] Event
- [Time] Event
- [Time] Event
## Root Cause
[Technical explanation of why this happened]
## Contributing Factors
- [Factor 1]
- [Factor 2]
## Resolution
[How the issue was resolved]
## Impact
- Users affected: [Number]
- Revenue impact: [If applicable]
- Data impact: [If applicable]
## Lessons Learned
### What went well
- [Item]
### What could be improved
- [Item]
## Action Items
| Action | Owner | Due Date | Status |
| -------- | ------ | -------- | -------- |
| [Action] | [Name] | [Date] | [Status] |
## Attendees
[List of postmortem meeting attendees]
Quick Reference Card#
Emergency Contacts#
| Role | Contact |
|---|---|
| On-Call | oncall@tara.app |
| Engineering | engineering@tara.app |
| Database Admin | dba@tara.app |
| Security | security@tara.app |
Rollback Commands#
# Vercel (API/Web)
vercel rollback [URL] --token $VERCEL_TOKEN
# AWS ECS
aws ecs update-service --cluster tara-production \
--service tara-api --task-definition tara-api:PREVIOUS
# Database Migration
npm run db:rollback -- --step 1
# Feature Flag Disable
curl -X POST https://api.tara.app/admin/flags \
-H "Authorization: Bearer $ADMIN_TOKEN" \
-d '{"flag": "feature_name", "enabled": false}'
Verification Endpoints#
# API Health
curl https://api.tara.app/health
# Web Health
curl https://tara.app
# Database Connection
curl https://api.tara.app/health/db
Last Updated: January 2026
For questions about rollback procedures, contact ops@tara.app