# V2 Asset Backup And Disaster Recovery

This guide covers `TODOS.phase-72.72.20.1.12`: scheduled snapshots of Git LFS
asset storage and geographic replication for disaster recovery.

## Source Of Truth

- Manifest: `V2/ue/Build/AssetPipeline/v2-asset-backup-dr.json`
- Tool: `V2/ue/Tools/v2_asset_backup_dr.py`
- Validator: `V2/ue/Tools/check-v2-asset-backup-dr.py`
- Scheduled workflow: `.github/workflows/v2-asset-backup-dr.yml`
- Plan output: `V2/ue/Saved/CI/AssetBackupDRPlan.json`

## Snapshot Schedule

The backup plan runs every six hours with cron `15 */6 * * *`. The plan captures
versioned LFS objects, LFS lock state, a pointer inventory, and a SHA-256
checksum manifest. Source deletion is explicitly disabled; backup jobs only copy
from the primary LFS bucket.

## Geographic Replication

The primary LFS backend is `oshun-v2-lfs-assets` in `us-east-1`. Each snapshot
is planned to replicate to `us-west-2` and `eu-central-1` replica buckets using
the `oshun-v2-lfs-assets-{region}-backup` naming convention. Replicas retain
object versions, object-lock semantics, and AES256 server-side encryption.

## Restore Drills

Restore drills run from the latest snapshot pointer, copy replica content into a
scratch prefix, then validate with:

```bash
git lfs fsck
git lfs ls-files --all --long
python3 V2/ue/Tools/check-v2-asset-deduplication.py
python3 V2/ue/Tools/check-v2-asset-size-budget.py
```

The RPO target is six hours and the RTO target is four hours. Promoting a
replica to production requires manual approval so a failed region cannot rewrite
the active Git LFS endpoint automatically.

## Planning Commands

Generate and verify the DR plan:

```bash
python3 V2/ue/Tools/v2_asset_backup_dr.py plan \
  --json-report V2/ue/Saved/CI/AssetBackupDRPlan.json

python3 V2/ue/Tools/v2_asset_backup_dr.py verify \
  --plan-json V2/ue/Saved/CI/AssetBackupDRPlan.json
```

The tool emits executable storage commands for the backup runner, but CI treats
the plan as dry-run evidence unless storage credentials are supplied by the
scheduled backup environment.

## Validation Commands

```bash
python3 V2/ue/Tools/v2_asset_backup_dr.py self-test
python3 V2/ue/Tools/check-v2-asset-backup-dr.py
python3 -m json.tool V2/ue/Build/AssetPipeline/v2-asset-backup-dr.json
python3 V2/ue/Tools/check-v2-ci-workflow.py
```