Date: 2026-04-27 Decision driver:
aja-descope-decision.md (V1-P2-0331) Status:
V1 posture locked; V2 candidate set documented for the upcoming GPU pipeline
decision in V1-P2-0334.
This document records the V1 Kubernetes runtime posture for Aja and the V2 candidate set. It does not decide the V2 outcome — that requires the GPU pipeline call (V1-P2-0334) and budget input outside this audit's scope.
V1 posture — locked#
What runs on Kubernetes in V1#
| Workload | Source | Runtime shape |
|---|---|---|
svc-motion-pipeline |
apps/aja/svc-motion-pipeline |
Single Fastify Deployment behind a ClusterIP Service. CPU-only. Built on @lilith/fastify-core (same framework family already used by lilith-service). |
That is the entire V1 K8s footprint for Aja. Nothing else from apps/aja/ or
libs/aja/ runs as a cluster workload in V1:
apps/aja/cliis a developer/operator command, not a workload.apps/aja/svc-motion-aiis V2-deferred (descope decision).apps/aja/svc-reference-videois a skeleton; not on the V1 ship list.libs/aja/distributed-workers,batch-inference,pipeline-parallelism,model-optimizationare V2-deferred — none of their simulated K8s client / NVML / CUDA paths are activated.
Helm chart#
Create one chart, deploy/helm/aja-service/, modeled on the existing
deploy/helm/lilith-service/ chart (which already ships Chart.yaml +
templates/{deployment,configmap,service,ingress,hpa,pdb,networkpolicy}.yaml
- per-environment values files). The Aja chart is a strict subset:
- Resources: Deployment, Service, ConfigMap, HPA (CPU-based), PDB, NetworkPolicy. No DaemonSet. No StatefulSet. No GPU node selector.
- Replicas: start at
replicas: 2with HPA range[2, 6]keyed on CPU utilization. The orchestration service is stateless; horizontal scaling is fine. - Resources requests: modest CPU-only. Suggested starting point —
requests: { cpu: 200m, memory: 256Mi },limits: { cpu: 1000m, memory: 1Gi }. Tune in staging before promotion. - Storage: none. The service streams uploads to object storage and job state lives in Postgres / Redis (already provisioned by other charts).
- Secrets: OAuth2 client credentials, object-storage signed-URL signing key,
webhook-signing key. Inject via existing platform ExternalSecret operator
pattern (used by
lilith-service). - Ingress: internal only in V1 (ClusterIP + ingress-controller route
aja.<env>.internal). Public exposure waits on external API governance signoff.
The chart should not include any GPU-related fields (nvidia.com/gpu resource
requests, runtimeClassName: nvidia, GPU node selectors, nvidia-device-plugin
tolerations). Adding them would imply we have a GPU node pool, which we do not
in V1.
What is explicitly not in V1#
The following pieces would have to be shipped together to support the deferred ML libraries; none of them are in V1:
- GPU node pool. Cluster-config does not provision GPU nodes today.
- NVIDIA Device Plugin / GPU Operator. Not installed.
- Worker DaemonSet / Job-based inference workers. The
distributed-workerslib defines K8s client interfaces but its health checks are simulated (fault-tolerance.ts,pipeline-parallelism/src/multi-gpu.ts— see readiness audit §2.2). - GPU node autoscaler. Cluster Autoscaler is configured for the CPU pool today; GPU-aware autoscaling needs a separate config and a vendor decision (see V1-P2-0334).
- CRDs / operators for inference jobs. Not authored.
CI / release#
- No
aja-ci.ymlGitHub Actions workflow ships in V1. Aja is covered by the genericoshun-ci.yml(which runsvitestfor the libraries and apps that opt in viaproject.json). Add an Aja-specific workflow only when V2 brings GPU paths that need their own runners. - ArgoCD: add a single Application manifest pointing at the new
deploy/helm/aja-service/chart. Mirror thelilith-serviceArgo application spec (auto-sync disabled in production until SLOs are measured for two weeks).
V2 candidates (feeding V1-P2-0334)#
When V1-P2-0334 selects a GPU pipeline, the K8s runtime grows to host inference workers. The candidate set follows; no choice is made here.
Candidate A — In-cluster GPU pool with NVIDIA stack#
- GPU node pool (
g5.xlarge/a10g/ equivalent) provisioned via the existing terraform / cluster-config plane. - NVIDIA GPU Operator installs Device Plugin, container toolkit, NVML exporter.
- Inference workers run as a DaemonSet (one per GPU) or as a Deployment with
nvidia.com/gpu: 1requests. - Pros: single cluster, single observability surface, the cleanest fit for
distributed-workers's already-typed K8s client. - Cons: GPU nodes are expensive at idle; needs a cluster autoscaler policy that scales to zero outside business hours; ops complexity (driver upgrades, MIG / time-slicing decisions).
Candidate B — Cloud-managed inference (out of cluster)#
- Inference moves to a managed service (e.g. SageMaker, Vertex AI, or a serverless GPU provider). Aja's K8s footprint stays CPU-only and the orchestration service routes inference requests to the managed endpoint.
- Pros: zero GPU ops in the K8s cluster; managed scale-to-zero; predictable per-request cost.
- Cons: cold-start latency on serverless tiers; tighter coupling to a vendor; data-egress costs for video assets; identity boundary (cluster service accounts ↔ cloud IAM) needs a federated-identity setup.
Candidate C — Hybrid#
- CPU-only orchestration in K8s (V1 posture).
- GPU inference workers run on a sibling cluster or VM pool managed outside the primary K8s footprint, fronted by a private API.
- Pros: keeps GPU ops decoupled; allows iterating on GPU strategy without redeploying the orchestration service.
- Cons: another control plane; extra network hop; requires careful health/timeout tuning between control plane and inference plane.
Decision inputs needed for V1-P2-0334#
The GPU pipeline decision should be informed by:
- Volume estimate. Expected sustained inference QPS for V2 ML libs (currently unknown — needs product input).
- Latency budget. Coaching overlay is interactive; demonstration plans are not. Latency targets per endpoint determine whether serverless cold starts are acceptable.
- Vendor lock-in tolerance. Candidate B and C diverge on this.
- Existing cluster reuse. If the platform already operates a GPU pool for
another domain (e.g.
kalikaorfreyarendering), that tilts toward Candidate A. - Cost model under low utilization. GPUs at ~10 % utilization are the worst case for Candidate A; Candidates B/C amortize idle cost automatically.
Non-goals of this document#
- Does not pick between Candidates A / B / C. That belongs to V1-P2-0334.
- Does not commit to specific node types, instance families, vendor contracts, or budgets. Those follow the candidate selection.
- Does not change any other domain's K8s footprint (
lilith-service,saraswati-service).
Knock-on tasks#
deploy/helm/aja-service/chart authoring (separate ticket, gated on this strategy doc).- ArgoCD application manifest for
aja-service. aja-runbook.mdcovering deployment, on-call, rollback, and the "service is healthy but no jobs progress" failure mode (which in V1 almost always means object storage or webhook delivery is the culprit, not Aja itself).