# ADR-0004: RunPod Endpoint Provisioning Mechanism

**Status**: Accepted  
**Date**: 2026-02-22  
**Authors**: Isis DevOps, Isis Runtime Engineering  
**Reviewers**: `@oshun/devops`, `@oshun/platform`, `@oshun/ml-team`,
`@oshun/core-team`

## Context and Problem Statement

Isis requires desired-state lifecycle control for RunPod serverless endpoints.
Current state shows:

- endpoint credentials/config are managed in Terraform secrets
  (`infra/terraform/secrets/runpod.tf`)
- image deployment and endpoint mutation are already performed through a GitHub
  Actions workflow (`.github/workflows/deploy-runpod.yml`)
- there are no RunPod endpoint creation/update/delete resources in Terraform
  under `infra/terraform`

The remediation backlog requires a single provisioning mechanism decision before
implementing lifecycle automation (`F08-001`).

## Decision

Adopt an **API-driven CI control plane** as the canonical RunPod endpoint
provisioning mechanism.

Provisioning, updates, and reconciliation will run via repository-owned CI jobs
calling the RunPod GraphQL/API surface, with endpoint desired state committed
in-code and validated in CI.

## Decision Drivers

- Existing operational path already mutates endpoints through CI
  (`deploy-runpod.yml`), so this is the lowest-risk production-hardening path.
- Terraform in this repo currently governs AWS primitives and secret
  distribution, not RunPod endpoint lifecycle resources.
- A CI-native approach supports staged rollouts, drift detection, and reconcile
  loops with explicit release gating.
- It avoids introducing a second control plane while the RunPod provider surface
  and organizational workflows are still evolving.

## Considered Options

### Option A: Terraform-native RunPod provider/module (Rejected for now)

Pros:

- Familiar desired-state workflow.
- Existing Terraform plan/apply governance.

Cons:

- No established RunPod endpoint resource usage in this codebase.
- Would introduce a parallel control plane and migration complexity.

### Option B: API-driven CI jobs with declarative endpoint manifests (Chosen)

Pros:

- Extends existing, proven deployment workflow.
- Keeps RunPod operations in one automation surface.
- Enables progressive rollout and reconcile logic without blocking on provider
  maturity.

Cons:

- Requires disciplined script/runtime quality and strong CI safety checks.
- Drift handling must be explicitly implemented (not inherited from Terraform
  state).

### Option C: External control plane service/operator (Deferred)

Pros:

- Strong separation and potentially richer runtime orchestration.

Cons:

- Highest implementation and operational overhead for current scope.
- Unnecessary before F08 lifecycle/drift controls are fully delivered.

## Consequences

### Positive

- Unifies RunPod lifecycle operations under a single, auditable automation path.
- Preserves Terraform ownership of secrets/credential distribution.
- Provides clear sequencing for `F08-002` to `F08-008`.

### Tradeoffs

- CI reliability and script correctness become critical path for endpoint
  lifecycle.
- Additional validation and drift checks must be authored explicitly.

## Implementation Rules

- Terraform remains source-of-truth for RunPod credentials and endpoint secret
  publication.
- CI automation becomes source-of-truth executor for endpoint
  create/update/delete/reconcile.
- Endpoint desired state must be committed in-repo and validated pre-deploy.
- Any manual dashboard mutation is treated as drift and must be reconciled by
  automation.

## Rollout Sequence

1. Implement endpoint lifecycle automation in CI (`F08-002`).
2. Add in-repo desired-state definitions (`F08-003`).
3. Add drift detection + reconcile commands (`F08-004`, `F08-005`).
4. Add CI validation and staged rollout controls (`F08-006`, `F08-007`).
5. Publish outage/cost runbook (`F08-008`).

## Ticket Traceability

- `F08-001`
- `F08-002`
- `F08-003`
- `F08-004`
- `F08-005`
- `F08-006`
- `F08-007`
- `F08-008`

## References

- `docs/domains/isis/isis-domain-completeness-audit-2026-02-21.md`
- `docs/domains/isis/adr/ADR-0003-runpod-serverless-execution-strategy.md`
- `.github/workflows/deploy-runpod.yml`
- `infra/terraform/secrets/runpod.tf`
