# IaC & CI/CD Audit + Cost Optimization — 2026-07-04

**Directive.** Audit and optimize all Infrastructure-as-Code and CI/CD; make
recurring cost as low as possible until there are customers and scale. Plus
three architecture consolidations requested mid-audit:

1. **Move all compute from AWS EKS to AWS ECS Fargate, fully.**
2. **GPUs run on RunPod, never EC2.**
3. **Move web/admin hosting from Vercel to AWS Amplify.**

Result: a single-cloud AWS footprint that costs **well under $100/month for a
running staging environment** (and near-zero when parked), down from a
provision-everything design that would have run **several thousand $/month** the
day someone ran `terraform apply`.

Nothing is deployed yet, so the only *live* spend today is GitHub Actions
minutes on a private repo — cut hard below. The IaC changes make the first
`apply` cheap-by-default rather than a large bill.

---

## 1. Target architecture (after)

| Concern              | Before                                   | After                                                        |
| -------------------- | ---------------------------------------- | ------------------------------------------------------------ |
| CPU compute          | EKS + managed node groups                | **ECS Fargate** (Fargate Spot default), scale-to-zero capable |
| GPU compute          | psyche EKS GPU nodes (g5, on-demand, ×2) | **RunPod** (off-cluster, pay-per-second, $0 idle)            |
| Web / admin hosting  | Vercel                                   | **AWS Amplify Hosting** (Next.js SSR, WEB_COMPUTE)           |
| Ingress              | per-service ELBs / ingress controller    | one **shared ALB** + Cloud Map service discovery            |
| Service registry     | Kubernetes DNS                           | **AWS Cloud Map** private DNS (`oshun-<env>.internal`)      |
| Deploy mechanism     | Helm + ArgoCD GitOps                     | `deploy-ecs.yml` (backend) + `oshun-web-deploy.yml` (Amplify)|
| Canonical IaC        | ambiguous (`deploy/` vs `infrastructure/`)| **`infrastructure/terraform`** (single root module)         |

Compute is now serverless end to end: no EKS control-plane fee, no idle nodes,
no Karpenter, no EC2. GPU is entirely off-cluster on RunPod.

---

## 2. Cost model (estimates, on-demand us-east-1 list prices)

### 2.1 Platform, per environment

| Line item              | Before (as-written)                         | After (pre-customer default)                    |
| ---------------------- | ------------------------------------------- | ----------------------------------------------- |
| Control plane          | EKS $73/mo                                   | ECS $0                                           |
| Compute                | m6i.large ×3 on-demand ≈ $210/mo            | 3 Fargate Spot tasks @0.25vCPU/0.5GB ≈ **$8/mo** |
| Database               | db.t3.medium + 100GB ≈ $58/mo               | db.t4g.micro + 20GB ≈ **$14/mo**                |
| Cache                  | cache.t3.medium (×3 prod) ≈ $50–149/mo      | cache.t4g.micro ×1 ≈ **$12/mo**                 |
| NAT                    | per-AZ ×3 (prod) ≈ $96/mo                    | single NAT all envs ≈ **$32/mo**                |
| Ingress                | multiple ELBs                               | one ALB ≈ **$16/mo**                            |
| Web hosting            | Vercel plan                                 | Amplify free tier ≈ **$0**                       |
| **Per env**            | **≈ $420–580/mo**                           | **≈ $84/mo** (≈ $60 parked at desired_count 0)  |

### 2.2 psyche (realtime) GPU

`enable_gpu_nodes` / `enable_karpenter` defaulted **on** across dev/staging/prod:
`g5.2xlarge` ON_DEMAND × desired 2 ≈ **$1,469/mo** each environment, plus
Karpenter-managed general nodes and a second EKS control plane. Now defaulted
**off** — GPU inference goes to RunPod ($0 when idle). **~$1,500/mo eliminated
per psyche env.**

### 2.3 Headline

- **Before**, applying platform (3 envs) + one psyche env with GPU:
  order of **$3,000–5,000+/mo**.
- **After**, running one staging env, GPU on RunPod idle: **< $100/mo**;
  services parked (`desired_count = 0`) trims to the ALB + NAT + tiny DB/cache
  floor (~$60/mo). Scale is a variable flip, not a re-architecture.

### 2.4 CI (live spend today)

26 scheduled workflows ran daily/6-hourly on a private repo (paid minutes), one
on **windows-latest** (2× multiplier), several e2e/DAST/drift jobs against
environments that **do not exist yet**. Rough burn ≈ **$90–120/mo** of Actions
minutes for zero signal. After the cuts (below): **~$3–5/mo**.

---

## 3. What changed

### 3.1 New Terraform (canonical `infrastructure/terraform`)

- **`modules/ecs-fargate`** — cluster, FARGATE + FARGATE_SPOT capacity providers
  (Spot-weighted default, on-demand base 0 pre-customer), shared internet-facing
  ALB (HTTP-only until an ACM cert is supplied, then HTTPS + redirect), Cloud
  Map private DNS namespace, ALB + task security groups, task-execution role
  (ECR pull, logs, scoped SSM/Secrets), base task role.
- **`modules/ecs-service`** — per service: task definition (256/512 default,
  X86_64 to match CI, arch is a one-line var), service with deployment circuit
  breaker + rollback, optional ALB target group + listener rule (web services),
  Cloud Map registration, CPU-target autoscaling, `ignore_changes` on
  task_definition/desired_count so CI owns rollouts.
- **`modules/amplify`** — one Amplify app per front-end, pnpm-monorepo build
  spec, per-branch environments, optional custom domain. Auto-build off; CI
  drives releases.
- **`main.tf` / `variables.tf` / `outputs.tf`** rewritten: dropped the
  kubernetes/helm providers and the EKS module; added ECR repos (with lifecycle
  expiry) per service, CloudWatch log groups, the ECS cluster, a `services` map
  (bff = ALB-routed; content-service, telegram-bot = internal), RDS/Redis with
  Graviton micro defaults + single-AZ, S3, and the Amplify apps (web, admin).
  Single NAT for all envs; two AZs. `terraform validate` passes; tree is
  `fmt`-clean.
- **Deleted** `modules/eks/`.

### 3.2 Workflows

- **`deploy-ecs.yml`** rewritten for the V1 Fargate model: 3 real services,
  build via `docker/Dockerfile.node` (APP_NAME/APP_PATH), names aligned exactly
  to Terraform (cluster `oshun-<env>-ecs`, service `<name>`, family
  `<env>-<name>`, log group `/ecs/oshun-<env>/<name>`, roles constructed from
  account id), register-or-update task def, roll + wait-stable, prod approval
  gate. Fixes prior hard failures (it targeted `oshun-staging-lilith-api`,
  port 3000, cluster `oshun-staging` — none of which the platform creates).
- **`oshun-web-deploy.yml`** rewritten Vercel → Amplify: quality gate →
  `aws amplify start-job` (RELEASE) → poll → health check, prod approval gate.
- **`terraform.yml`** rewritten: it targeted the nonexistent `infra/terraform/**`
  with a per-module matrix. Now: `fmt -check` + `validate` as the always-on PR
  gate on `infrastructure/terraform`, plan/apply on dispatch via OIDC + env S3
  backend, prod apply behind an environment gate.
- **Deleted** `deploy.yml` (K8s/Helm CD — only consumer of the retired charts)
  and `oshun-web-preview.yml` (Vercel PR previews).

### 3.3 CI cost cuts

- **19 scheduled workflows disabled** (schedule commented, `workflow_dispatch`
  kept) — all targeted environments that don't exist yet: every e2e suite,
  DAST, a11y-nightly, runpod/iris drift detection, model-sync, the 6-hourly
  v2 asset-backup, and the nightly V2–V5 game builds (incl. the windows-latest
  one). Grep marker: `COST(pre-launch): scheduled run disabled`.
- **5 source-scanners reduced daily → weekly** (ci, iris-security,
  security-deps-and-secrets, test-coherence-check, adversarial-grep-trend);
  codeql + benchmarks were already weekly.
- Added `concurrency` to `release.yml`; `retention-days: 14` to 10 artifact
  uploads that defaulted to 90 days; bumped `softprops/action-gh-release@v1`→`v2`
  (EOL node16).

### 3.4 Docs

`V1/DEPENDENCIES.md` §20 + §5 and `V1/planning/SLO_AND_DR.md` §5 updated
(EKS→Fargate, Vercel→Amplify, GPU→RunPod, canonical tree). `DEPRECATED.md`
added to `infrastructure/helm`, `infrastructure/argocd`, `deploy/terraform`,
`deploy/argocd`.

---

## 4. Scale-up levers (flip these when customers arrive)

| Lever                              | Where                                                        |
| ---------------------------------- | ----------------------------------------------------------- |
| On-demand Fargate floor            | `fargate_on_demand_base` (main.tf sets 1 for prod)          |
| Per-AZ NAT (HA egress)             | `single_nat_gateway = false` in main.tf vpc block           |
| RDS Multi-AZ standby               | `rds_multi_az = true`                                        |
| Redis replicas                     | `redis_num_nodes`                                            |
| Bigger service tasks               | `services["…"].cpu/memory/desired_count`                    |
| Graviton (−20%) compute            | `cpu_architecture = "ARM64"` once CI builds `linux/arm64`   |
| Container Insights                 | `enable_container_insights = true`                          |
| Restore nightly CI cadence         | uncomment the `schedule:` blocks / restore daily crons      |
| Third AZ                           | widen the `local.azs` slice                                 |

---

## 5. Follow-ups — both completed 2026-07-04

- **psyche EKS → Fargate (full) — DONE.** Psyche's three environments
  (dev/staging/prod) were migrated off EKS onto ECS Fargate, reusing the
  platform's `ecs-fargate` / `ecs-service` modules. Removed: the `eks` and
  `karpenter` modules, the kubernetes+helm providers, all ~9 in-cluster helm/k8s
  resources per env (ALB controller, ingress-nginx, external-dns, cert-manager,
  metrics-server, prometheus-adapter, fluent-bit, ingress classes, letsencrypt
  issuers), the IRSA `iam` module, and the psyche service Helm charts
  (`infrastructure/psyche/kubernetes/`). Added: `module.ecs` + 10 `ecs-service`
  instances (api-gateway ALB-routed; the rest internal via Cloud Map; avatar/
  voice call RunPod for GPU), an S3 task-role policy replacing the IRSA app role,
  a Route53 alias → ALB replacing external-dns, and a WAF→ALB association. The
  KMS key policy now grants the ECS task role. Cleaned 26-27 orphaned K8s
  variables per env. `psyche-deploy.yml` rewritten from kubectl/helm to the ECS
  build→register→roll model. All three envs `terraform validate` clean; the K8s
  add-ons collapse into native AWS (ALB/ACM/CloudWatch/Cloud Map). GPU stays on
  RunPod. Node-floor trims from the prior pass are subsumed (there are no nodes).

- **Physical deletion of the superseded K8s trees — DONE.** Removed
  `infrastructure/helm/*` (except `maya-orchestration`, which is Agones/K8s and
  stays), `infrastructure/argocd`, `infrastructure/kubernetes`,
  `deploy/kubernetes`, the lilith-platform `deploy/helm` + `deploy/argocd`, the
  dead `deploy/webrtc` (coturn/helm; TURN already lives in bellona-remote ECS),
  the psyche K8s charts, and the three dead lilith K8s deploy scripts
  (`generate-argocd-apps.sh`, `generate-helm-values.sh`, `promote-deployment.sh`,
  `deploy/promote-release.sh`) — while preserving the rest of `scripts/lilith`
  (test/docs/perf tooling a live benchmark imports). The blockers were resolved
  honestly, not gamed: `verify-phase-12-completion.mjs` was rewritten to verify
  the **current** ECS deployment consolidation (real checks + live
  `terraform validate`), `verify-phase-49`'s infra-evidence list was repointed
  to the ECS stack, and `TODOS/phase-12.md` carries a supersession note. Both
  phase verifications pass; a repo-wide sweep confirms **zero dangling
  references** to any deleted tree.
- **Physical deletion of the deprecated K8s charts — evaluated and rejected
  (2026-07-04).** The K8s trees stay deprecated-in-place, not deleted, because
  every substantial one is load-bearing for a gate or tool that deletion would
  break, and rewriting those gates to permit deletion would be gaming them:
    - `infrastructure/helm`, `infrastructure/argocd`, `infrastructure/kubernetes`
      → `scripts/verify-phase-12-completion.mjs` (and `verify-phase-49`) assert
      their existence and content as completed-phase gates.
    - `deploy/helm`, `deploy/argocd` → the lilith deployment tooling
      (`scripts/lilith/promote-deployment.sh`, `generate-helm-values.sh`,
      `generate-argocd-apps.sh`) reads them.
    - `infrastructure/helm/maya-orchestration/` → Maya's Agones fleets need K8s
      and its test asserts against it.
    - Only `deploy/kubernetes/` (4 generic RBAC/pod-security files) is
      unreferenced; left in place as it is moot-but-harmless under ECS.
  Conclusion: deprecation notices (each tree's `DEPRECATED.md`) are the correct
  terminal state; there is no safe physical deletion without collateral damage.
- **Global `timeout-minutes` sweep.** Most heavy workflows already set one;
  a blanket default risks failing legitimate long UE builds, so it needs
  per-workflow tuning rather than a mechanical pass.
- **Pre-existing actionlint findings** in untouched workflows (a few
  `[expression]`/`[syntax-check]` hits; the `[runner-label]` hits are valid
  self-hosted labels) — out of scope for this cost/migration pass.

---

## 6. Verification performed

- `terraform validate` — **passes**; `terraform fmt -check -recursive` — clean.
- `actionlint` across all 138 workflows — the files rewritten this session
  (deploy-ecs, oshun-web-deploy, terraform, release, ci) are clean.
- YAML parse check across every edited workflow — all parse; every
  schedule-edited file retains at least one trigger (no workflow left
  un-runnable).

---

## 7. V1 deployment inventory — completeness check (2026-07-04)

Verified every deployable V1 surface is covered. V1's platform = the "Oshun +
Metis workloads" that shared the EKS cluster (§20); the domain apps
(lilith/yemaya/isis/sophia/hathor/bellona/nyx/arete/veritas) are composed into
the Oshun BFF as libraries, not deployed as V1 services, and Tara / Iris /
Psyche deploy independently (Tara own Vercel+mobile; Iris own terraform; Psyche
migrated to its own Fargate above).

**Backend services → ECS Fargate (`deploy-ecs.yml`, all 5):**

| Service | Package | Path |
| --- | --- | --- |
| bff | @oshun/bff | apps/oshun/bff (ALB `/*`) |
| content-service | @oshun/content-service-app | apps/oshun/content-service |
| telegram-bot | @oshun/telegram-bot | apps/oshun/telegram-bot |
| metis-api-gateway | @metis/api-gateway | apps/metis/api-gateway (ALB `/metis/*`) |
| metis-worker | @metis/worker | apps/metis/worker |

**Front-ends → AWS Amplify (`oshun-web-deploy.yml`, all 6):**
`@oshun/web`, `@oshun/admin`, `@oshun/tenant-admin`, `@oshun/telegram-miniapp`,
`@metis/web`, `@metis/admin`.

**Data / ingress:** RDS Postgres, ElastiCache Redis, S3, one shared ALB, Cloud
Map — all in `infrastructure/terraform`. The old `deploy-ecs.yml` "common"
services (event-bus, cache, health-monitor, metrics-collector) map to
ElastiCache / CloudWatch / the ALB, not to separate Fargate tasks.

**Not part of the cloud migration (separate deploy channels — flagged, not
silently dropped):**

- **Mobile** (`apps/oshun/mobile`, `apps/oshun/admin-mobile`, `apps/metis/mobile`)
  ships via EAS to the app stores, not to the platform cluster.
  `oshun-mobile-release.yml` exists; **`apps/metis/mobile` has no release
  workflow** — a mobile-release gap, not an ECS/Amplify one.
- **Metis databases:** `metis-api-gateway`/`metis-worker` need their own
  Postgres DBs. `METIS_DATABASE_URL` is still undocumented in `.env.example`
  (pre-existing gap, DEPENDENCIES §21). The RDS instance exists; the metis
  database + secret must be provisioned before Metis serves traffic.
- **`apps/oshun/legal`** is static markdown (privacy-policy / terms-of-service),
  rendered by the web app — not a separate deployment.
- **`apps/oshun/clipper-extension`** publishes to browser extension stores.

---

## 8. Deployment consolidation — single plane (2026-07-04)

Follow-up to simplify V1 deployment. Two decisions: (1) **single hosting plane**
— everything on Fargate, drop Amplify; (2) **fold Psyche in** — one account,
cluster, VPC, DB, state.

**Front-ends → Fargate (Amplify removed).** The 6 Next.js apps now run as
containers behind the shared ALB, host-routed on `<host_prefix>.<platform_domain>`:
`web` (apex), `admin`, `tenant-admin`, `tg` (telegram-miniapp, static export),
`metis`, `metis-admin`. Both SSR (Next `output: 'standalone'` → `node server.js`)
and the one static export (telegram-miniapp) are built + served by
`docker/Dockerfile.web`. Deleted `modules/amplify`, the amplify vars, and
`oshun-web-deploy.yml`; added `platform_domain` + `host_prefix` routing.

**Psyche folded into the platform.** Its 10 services join the `services` map as
`psyche-*` (api-gateway ALB-routed; rest internal; GPU via RunPod). Deleted the
entire separate `infrastructure/psyche/terraform` tree (its own account/VPC/RDS/
Redis/ALB/state) and `psyche-deploy.yml`. Psyche's Python services build from
`services/psyche/<svc>/Dockerfile`.

**One deploy path.** `deploy-ecs.yml` now builds + deploys **every** V1 workload
(21 services: backend + front-ends + psyche) with per-service Dockerfile
resolution. `verify-phase-12/49` updated to the single-plane reality; the launch-
gate analytics manifests repointed off the deleted `oshun-web-deploy.yml` (deploy
evidence → `deploy-ecs.yml`, lighthouse → `oshun-web-lighthouse.yml`).

**Net:** one `terraform apply`, one deploy workflow, one cluster/VPC/DB/ALB, one
hosting model. Roughly halves standing infra (the duplicate psyche stack is gone).

**Build verification (done locally, 2026-07-05).** The images were actually
built and smoke-tested on-box (Docker + Node are available here), which caught
three real defects that would have broken the first CI deploy — all fixed:

1. **`.dockerignore` context bloat.** It excluded `node_modules/.git/.next` but
   not Rust `target/` dirs, the V2–V9 UE trees, or `.claude` worktrees, so the
   build context was **~56 GB** and blew up Docker's disk. Added those
   exclusions → context ~2 GB.
2. **`E2BIG` on install.** The huge root `pnpm` config (40+ overrides, 30+
   `onlyBuiltDependencies`, patches) is injected into every spawned lifecycle
   script's env, exceeding the OS arg limit in-container — so *every* scripted
   install failed instantly with no output. Added `--ignore-scripts` to the
   install in `Dockerfile.node` and `Dockerfile.web` (V1 services have no direct
   native deps; it also correctly skips the electron node-gyp that must not
   build on Node 22).
3. **Oversized SSR image.** The `next start` Dockerfile kept the whole build
   stage (multi-GB, monorepo `node_modules`) and filled the disk on export.
   Switched the SSR apps to Next `output: 'standalone'` (+ `outputFileTracingRoot`
   = monorepo root) served by `docker/Dockerfile.web` → a ~905 MB image; also
   fixed that Dockerfile's runner stage to promote `APP_PATH` to a runtime ENV
   (the standalone CMD needs it). Retired `Dockerfile.web-ssr`.

Verified: `@oshun/web` builds to a standalone image and the container boots
(Next.js 16, listening on `0.0.0.0:8080`, serves HTTP 307 — healthy under the
ALB's 200-399 matcher). The other 4 SSR apps use the identical config/Dockerfile
pattern.

**Still owed** (genuinely needs the cloud, not buildable here): host DNS/ACM
wiring for the front-end subdomains; the psyche Python image builds under CI (own
Dockerfiles); and provisioning the Metis + Psyche databases in the shared RDS.
Front-ends default to 0.25 vCPU / 512 MiB — Next.js SSR may want a memory bump
under load (per-service lever).