Status: Accepted
Date: 2026-02-22
Authors: Isis DevOps, Isis Runtime Engineering
Reviewers: @oshun/devops, @oshun/platform, @oshun/ml-team,
@oshun/core-team
Context and Problem Statement#
Isis requires desired-state lifecycle control for RunPod serverless endpoints. Current state shows:
- endpoint credentials/config are managed in Terraform secrets
(
infra/terraform/secrets/runpod.tf) - image deployment and endpoint mutation are already performed through a GitHub
Actions workflow (
.github/workflows/deploy-runpod.yml) - there are no RunPod endpoint creation/update/delete resources in Terraform
under
infra/terraform
The remediation backlog requires a single provisioning mechanism decision before
implementing lifecycle automation (F08-001).
Decision#
Adopt an API-driven CI control plane as the canonical RunPod endpoint provisioning mechanism.
Provisioning, updates, and reconciliation will run via repository-owned CI jobs calling the RunPod GraphQL/API surface, with endpoint desired state committed in-code and validated in CI.
Decision Drivers#
- Existing operational path already mutates endpoints through CI
(
deploy-runpod.yml), so this is the lowest-risk production-hardening path. - Terraform in this repo currently governs AWS primitives and secret distribution, not RunPod endpoint lifecycle resources.
- A CI-native approach supports staged rollouts, drift detection, and reconcile loops with explicit release gating.
- It avoids introducing a second control plane while the RunPod provider surface and organizational workflows are still evolving.
Considered Options#
Option A: Terraform-native RunPod provider/module (Rejected for now)#
Pros:
- Familiar desired-state workflow.
- Existing Terraform plan/apply governance.
Cons:
- No established RunPod endpoint resource usage in this codebase.
- Would introduce a parallel control plane and migration complexity.
Option B: API-driven CI jobs with declarative endpoint manifests (Chosen)#
Pros:
- Extends existing, proven deployment workflow.
- Keeps RunPod operations in one automation surface.
- Enables progressive rollout and reconcile logic without blocking on provider maturity.
Cons:
- Requires disciplined script/runtime quality and strong CI safety checks.
- Drift handling must be explicitly implemented (not inherited from Terraform state).
Option C: External control plane service/operator (Deferred)#
Pros:
- Strong separation and potentially richer runtime orchestration.
Cons:
- Highest implementation and operational overhead for current scope.
- Unnecessary before F08 lifecycle/drift controls are fully delivered.
Consequences#
Positive#
- Unifies RunPod lifecycle operations under a single, auditable automation path.
- Preserves Terraform ownership of secrets/credential distribution.
- Provides clear sequencing for
F08-002toF08-008.
Tradeoffs#
- CI reliability and script correctness become critical path for endpoint lifecycle.
- Additional validation and drift checks must be authored explicitly.
Implementation Rules#
- Terraform remains source-of-truth for RunPod credentials and endpoint secret publication.
- CI automation becomes source-of-truth executor for endpoint create/update/delete/reconcile.
- Endpoint desired state must be committed in-repo and validated pre-deploy.
- Any manual dashboard mutation is treated as drift and must be reconciled by automation.
Rollout Sequence#
- Implement endpoint lifecycle automation in CI (
F08-002). - Add in-repo desired-state definitions (
F08-003). - Add drift detection + reconcile commands (
F08-004,F08-005). - Add CI validation and staged rollout controls (
F08-006,F08-007). - Publish outage/cost runbook (
F08-008).
Ticket Traceability#
F08-001F08-002F08-003F08-004F08-005F08-006F08-007F08-008
References#
docs/domains/isis/isis-domain-completeness-audit-2026-02-21.mddocs/domains/isis/adr/ADR-0003-runpod-serverless-execution-strategy.md.github/workflows/deploy-runpod.ymlinfra/terraform/secrets/runpod.tf