# Aja Kubernetes Runtime Strategy (V1-P2-0333)

**Date:** 2026-04-27 **Decision driver:**
[`aja-descope-decision.md`](./aja-descope-decision.md) (V1-P2-0331) **Status:**
V1 posture **locked**; V2 candidate set documented for the upcoming GPU pipeline
decision in V1-P2-0334.

This document records the V1 Kubernetes runtime posture for Aja and the V2
candidate set. It does **not** decide the V2 outcome — that requires the GPU
pipeline call (V1-P2-0334) and budget input outside this audit's scope.

## V1 posture — locked

### What runs on Kubernetes in V1

| Workload              | Source                         | Runtime shape                                                                                                                                             |
| --------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `svc-motion-pipeline` | `apps/aja/svc-motion-pipeline` | Single Fastify Deployment behind a ClusterIP Service. CPU-only. Built on `@lilith/fastify-core` (same framework family already used by `lilith-service`). |

That is the entire V1 K8s footprint for Aja. Nothing else from `apps/aja/` or
`libs/aja/` runs as a cluster workload in V1:

- `apps/aja/cli` is a developer/operator command, not a workload.
- `apps/aja/svc-motion-ai` is V2-deferred
  ([descope decision](./aja-descope-decision.md)).
- `apps/aja/svc-reference-video` is a skeleton; not on the V1 ship list.
- `libs/aja/distributed-workers`, `batch-inference`, `pipeline-parallelism`,
  `model-optimization` are V2-deferred — none of their simulated K8s client /
  NVML / CUDA paths are activated.

### Helm chart

Create one chart, **`deploy/helm/aja-service/`**, modeled on the existing
`deploy/helm/lilith-service/` chart (which already ships `Chart.yaml` +
`templates/{deployment,configmap,service,ingress,hpa,pdb,networkpolicy}.yaml`

- per-environment values files). The Aja chart is a strict subset:

* **Resources:** Deployment, Service, ConfigMap, HPA (CPU-based), PDB,
  NetworkPolicy. No DaemonSet. No StatefulSet. No GPU node selector.
* **Replicas:** start at `replicas: 2` with HPA range `[2, 6]` keyed on CPU
  utilization. The orchestration service is stateless; horizontal scaling is
  fine.
* **Resources requests:** modest CPU-only. Suggested starting point —
  `requests: { cpu: 200m, memory: 256Mi }`,
  `limits: { cpu: 1000m, memory: 1Gi }`. Tune in staging before promotion.
* **Storage:** none. The service streams uploads to object storage and job state
  lives in Postgres / Redis (already provisioned by other charts).
* **Secrets:** OAuth2 client credentials, object-storage signed-URL signing key,
  webhook-signing key. Inject via existing platform ExternalSecret operator
  pattern (used by `lilith-service`).
* **Ingress:** internal only in V1 (ClusterIP + ingress-controller route
  `aja.<env>.internal`). Public exposure waits on external API governance
  signoff.

The chart should not include any GPU-related fields (`nvidia.com/gpu` resource
requests, `runtimeClassName: nvidia`, GPU node selectors, `nvidia-device-plugin`
tolerations). Adding them would imply we have a GPU node pool, which we do not
in V1.

### What is explicitly **not** in V1

The following pieces would have to be shipped together to support the deferred
ML libraries; none of them are in V1:

- **GPU node pool.** Cluster-config does not provision GPU nodes today.
- **NVIDIA Device Plugin / GPU Operator.** Not installed.
- **Worker DaemonSet / Job-based inference workers.** The `distributed-workers`
  lib defines K8s client interfaces but its health checks are simulated
  (`fault-tolerance.ts`, `pipeline-parallelism/src/multi-gpu.ts` — see
  [readiness audit §2.2](./aja-readiness-audit.md#22-mixed)).
- **GPU node autoscaler.** Cluster Autoscaler is configured for the CPU pool
  today; GPU-aware autoscaling needs a separate config and a vendor decision
  (see [V1-P2-0334](#v2-candidates-feeding-v1-p2-0334)).
- **CRDs / operators for inference jobs.** Not authored.

### CI / release

- No `aja-ci.yml` GitHub Actions workflow ships in V1. Aja is covered by the
  generic `oshun-ci.yml` (which runs `vitest` for the libraries and apps that
  opt in via `project.json`). Add an Aja-specific workflow only when V2 brings
  GPU paths that need their own runners.
- ArgoCD: add a single Application manifest pointing at the new
  `deploy/helm/aja-service/` chart. Mirror the `lilith-service` Argo application
  spec (auto-sync disabled in production until SLOs are measured for two weeks).

## V2 candidates (feeding V1-P2-0334)

When V1-P2-0334 selects a GPU pipeline, the K8s runtime grows to host inference
workers. The candidate set follows; **no choice is made here**.

### Candidate A — In-cluster GPU pool with NVIDIA stack

- GPU node pool (`g5.xlarge` / `a10g` / equivalent) provisioned via the existing
  terraform / cluster-config plane.
- NVIDIA GPU Operator installs Device Plugin, container toolkit, NVML exporter.
- Inference workers run as a DaemonSet (one per GPU) or as a Deployment with
  `nvidia.com/gpu: 1` requests.
- Pros: single cluster, single observability surface, the cleanest fit for
  `distributed-workers`'s already-typed K8s client.
- Cons: GPU nodes are expensive at idle; needs a cluster autoscaler policy that
  scales to zero outside business hours; ops complexity (driver upgrades, MIG /
  time-slicing decisions).

### Candidate B — Cloud-managed inference (out of cluster)

- Inference moves to a managed service (e.g. SageMaker, Vertex AI, or a
  serverless GPU provider). Aja's K8s footprint stays CPU-only and the
  orchestration service routes inference requests to the managed endpoint.
- Pros: zero GPU ops in the K8s cluster; managed scale-to-zero; predictable
  per-request cost.
- Cons: cold-start latency on serverless tiers; tighter coupling to a vendor;
  data-egress costs for video assets; identity boundary (cluster service
  accounts ↔ cloud IAM) needs a federated-identity setup.

### Candidate C — Hybrid

- CPU-only orchestration in K8s (V1 posture).
- GPU inference workers run on a sibling cluster or VM pool managed outside the
  primary K8s footprint, fronted by a private API.
- Pros: keeps GPU ops decoupled; allows iterating on GPU strategy without
  redeploying the orchestration service.
- Cons: another control plane; extra network hop; requires careful
  health/timeout tuning between control plane and inference plane.

### Decision inputs needed for V1-P2-0334

The GPU pipeline decision should be informed by:

1. **Volume estimate.** Expected sustained inference QPS for V2 ML libs
   (currently unknown — needs product input).
2. **Latency budget.** Coaching overlay is interactive; demonstration plans are
   not. Latency targets per endpoint determine whether serverless cold starts
   are acceptable.
3. **Vendor lock-in tolerance.** Candidate B and C diverge on this.
4. **Existing cluster reuse.** If the platform already operates a GPU pool for
   another domain (e.g. `kalika` or `freya` rendering), that tilts toward
   Candidate A.
5. **Cost model under low utilization.** GPUs at ~10 % utilization are the worst
   case for Candidate A; Candidates B/C amortize idle cost automatically.

## Non-goals of this document

- **Does not pick** between Candidates A / B / C. That belongs to V1-P2-0334.
- **Does not commit** to specific node types, instance families, vendor
  contracts, or budgets. Those follow the candidate selection.
- **Does not change** any other domain's K8s footprint (`lilith-service`,
  `saraswati-service`).

## Knock-on tasks

- `deploy/helm/aja-service/` chart authoring (separate ticket, gated on this
  strategy doc).
- ArgoCD application manifest for `aja-service`.
- `aja-runbook.md` covering deployment, on-call, rollback, and the "service is
  healthy but no jobs progress" failure mode (which in V1 almost always means
  object storage or webhook delivery is the culprit, not Aja itself).
