# Aja GPU Pipeline Strategy (V1-P2-0334)

**Date:** 2026-04-27 **Decision drivers:**
[`aja-readiness-audit.md`](./aja-readiness-audit.md) (V1-P2-0330),
[`aja-descope-decision.md`](./aja-descope-decision.md) (V1-P2-0331),
[`aja-runtime-strategy.md`](./aja-runtime-strategy.md) (V1-P2-0333). **Status:**
**V1 posture locked** (no GPU pipeline in V1). V2 candidate matrix recorded;
final selection deferred to a V2 ticket once the inputs identified below are
gathered.

## Decision

**V1: no GPU pipeline.** Aja ships only the CPU-only orchestration service in
V1, exactly as locked in V1-P2-0331 and V1-P2-0333. None of the GPU-related
libraries (`batch-inference` GPU memory paths, `pipeline-parallelism`,
`model-optimization`, `neural-retargeting`, `human-mesh-recovery`,
`pose-lifting`, `depth-sensing`, `multi-view-reconstruction`,
`distributed-workers`) are wired into a real runtime in V1.

**V2: pick from the candidate matrix below once the four decision inputs are
gathered.** This document does not pick — picking now without volume, latency,
and budget data would be premature.

## Why no GPU pipeline in V1

Recap from upstream tasks:

- The readiness audit found 17 explicit `"in production"` markers and 124
  `Math.random()` calls in implementation source — load-bearing GPU paths are
  simulated. Lighting them up needs real models, drivers, and schedulers, none
  of which are present.
- SMPL model weights, ONNX runtime selection, and a GPU operator are all
  out-of-monorepo decisions that the descope decision (V1-P2-0331) pulled out of
  V1's critical path.
- The K8s strategy (V1-P2-0333) ships zero GPU node pool. There is literally
  nowhere for GPU code to run in V1 even if it were finished.

Shipping V1 without a GPU pipeline is therefore consistent and honest: there is
no half-built fallback the user could accidentally hit.

## V2 candidate matrix

Each candidate trades operational complexity against cost, latency, and lock-in.
The aim of this section is to give the V2 owner a side-by-side comparison they
can decide from once the input section below is filled in.

### A — In-cluster CUDA on NVIDIA GPUs (via NVML / CUDA Runtime)

- **Stack:** NVIDIA GPU Operator on the same K8s cluster as
  `svc-motion-pipeline`. Inference workers run as Deployments / Jobs with
  `nvidia.com/gpu: 1` requests, scheduled to a GPU node pool. Health and
  utilization come from NVML exporters; the simulated paths in
  `pipeline-parallelism/src/multi-gpu.ts` (lines 89/219/313/413/442/498/689, see
  [audit §2.2](./aja-readiness-audit.md#22-mixed)) get real implementations.
- **Inference runtime:** ONNX Runtime with CUDA execution provider, or TensorRT
  for the model-optimization library. Either is broadly supported by the
  published model methods named in the Aja README (HMR 2.0, CLIFF, PyMAF-X,
  4D-Humans, WHAM).
- **Pros:** unified observability and identity story (same cluster, same service
  mesh). Already-typed `distributed-workers` Kubernetes client fits this
  directly. Lowest data-egress cost — video assets stay in region.
- **Cons:** GPU nodes idle expensively. Cluster Autoscaler must support
  scale-to-zero for the GPU pool to be cost-effective. Driver upgrades and MIG /
  time-slicing decisions add ops surface area.

### B — ROCm on AMD GPUs

- **Stack:** Same shape as A but on AMD Instinct GPUs (MI200/MI300) via ROCm.
  Possible if the platform already operates AMD nodes for another domain.
- **Pros:** typically cheaper per-FLOP than NVIDIA in the same tier;
  diversification away from a single GPU vendor.
- **Cons:** smaller ecosystem of pretrained models that ship with ROCm builds.
  Several ML methods named in the Aja README do not have first-party ROCm
  support — bringing them up means recompiling kernels and validating numerics.
  Driver / runtime maturity (RCCL, MIOpen) lags CUDA. Most ONNX Runtime
  providers are CUDA-first.
- **Verdict if no AMD GPUs already exist on platform:** rule out. The porting
  cost is non-trivial and the V2 ML stack already targets CUDA.

### C — Cloud-managed inference (SageMaker / Vertex AI / serverless GPU)

- **Stack:** Inference moves out of the K8s cluster entirely.
  `svc-motion-pipeline` posts inference requests over an SDK / REST boundary to
  the managed endpoint. Models are deployed via the cloud provider's model
  registry. The K8s footprint stays exactly the V1 CPU-only shape.
- **Pros:** zero GPU ops on the platform team. Managed scale-to-zero.
  Per-request billing — predictable cost when load is bursty. Fastest way to get
  V2 inference live.
- **Cons:** cold-start latency on serverless tiers (10s–30s on first request) is
  incompatible with the coaching-overlay use case (interactive cues; latency
  budget likely <2 s). Vendor lock-in pulls model weights, monitoring, and
  identity into the chosen cloud. Data-egress costs scale with video volume.
  Cross-cloud federated identity needed if the primary K8s cluster lives
  elsewhere.

### D — Hybrid (CPU control plane + sibling GPU plane)

- **Stack:** `svc-motion-pipeline` stays where V1 puts it (CPU K8s). GPU workers
  run as a separate VM pool / sibling cluster behind a private API. The
  `distributed-workers` lib's K8s client is rewritten as an HTTP / gRPC client.
- **Pros:** decouples GPU-ops from the platform K8s. Lets the team iterate on
  the GPU plane without coordinating cluster releases. Easiest path to running
  on bare-metal GPU servers if cost demands it.
- **Cons:** extra network hop adds latency. Two control planes to observe.
  Health / timeout / retry semantics must be tuned across the boundary or
  transient blips become job failures.

## Decision matrix

| Criterion                                                           | A: in-cluster CUDA | B: ROCm |     C: cloud-managed     |    D: hybrid     |
| ------------------------------------------------------------------- | :----------------: | :-----: | :----------------------: | :--------------: |
| Tracks the existing typed K8s client in `distributed-workers`       |      **Yes**       |   Yes   |        No (HTTP)         |    No (HTTP)     |
| Coaching-overlay latency budget achievable                          |        Yes         |   Yes   |  **Risky** (cold start)  | Yes if always-on |
| Idle-cost behavior at <10 % utilization                             |        Bad         |   Bad   | **Best** (scale-to-zero) |       Mid        |
| Vendor lock-in                                                      |        Mid         |   Mid   |         **High**         |       Mid        |
| Data-egress cost for video assets                                   |        Low         |   Low   |         **High**         |     Low–mid      |
| Driver / runtime maturity for Aja-named methods (HMR/CLIFF/PyMAF-X) |      **Best**      |   Mid   |      Best (managed)      |       Best       |
| New ops surface for the platform team                               |        Mid         |   Mid   |        **Lowest**        |     Highest      |

(**bold** marks the candidate that wins on that row; ties allowed.)

No candidate dominates. The selection is sensitive to the four inputs below.

## Decision inputs the V2 owner must collect

These are the same inputs flagged in the runtime strategy doc, restated here as
concrete asks:

1. **Sustained inference QPS estimate** — what does the V2 product line actually
   need? Coaching-overlay traffic profile is the hardest case (high-frequency,
   low-latency). Demonstration plans are batchable.
2. **Per-endpoint latency budget** — coaching-overlay needs sub-2 s P95;
   demonstration plans tolerate seconds. The latency target is the main driver
   against Candidate C's serverless tier.
3. **Existing GPU footprint on the platform** — if Aja can ride an
   already-funded GPU pool (e.g. `kalika` rendering, `freya` voice synthesis),
   Candidate A becomes nearly free incremental cost. If not, Candidates C and D
   become more attractive at low utilization.
4. **Vendor / lock-in tolerance** — Candidate C ties model weights and
   monitoring to a single cloud. If the broader org wants multi-cloud / on-prem
   optionality, Candidate C is excluded.

If all four inputs land "low volume, relaxed latency, no existing GPU pool,
lock-in OK," the right call is **C**. If they land "high volume, tight latency,
existing GPU pool, lock-in not OK," the right call is **A**. Anything in between
favors **D**.

## Cross-references for the V2 ticket

- `libs/aja/distributed-workers/src/fault-tolerance.ts` — simulated K8s health
  checks; replace with real client calls for Candidates A or D.
- `libs/aja/pipeline-parallelism/src/multi-gpu.ts` — 7 simulated CUDA / NVML
  paths; gate behind a feature flag and only enable for Candidate A.
- `libs/aja/model-optimization/src/tensorrt-compiler.ts:267` — simulated
  TensorRT engine builds; the real implementation is Candidate-A specific.
- `libs/aja/human-mesh-recovery/src/smpl-recovery.ts:48` — placeholder SMPL
  data; needs licensed SMPL/SMPL-X model files. This is an out-of-monorepo
  procurement step that V2 must complete regardless of the candidate chosen.
- `libs/aja/neural-retargeting/src/inference.ts` — typed ONNX interfaces but no
  `onnxruntime` dependency; V2 must add it (CUDA EP for Candidate A, CPU EP for
  fallback in D's control plane, or remove entirely if Candidate C is picked and
  the cloud provider's SDK is used directly).

## Non-goals of this document

- Picks no candidate. The V2 owner picks once decision inputs are in.
- Does not commit to specific instance families, GPU counts, or vendor
  contracts.
- Does not change V1 — V1's posture (no GPU pipeline) is already locked in
  V1-P2-0331 and V1-P2-0333.
