Status: Accepted
Date: 2026-02-21
Authors: Isis ML Platform, Isis Runtime Engineering
Reviewers: @oshun/ml-team, @oshun/devops, @oshun/platform, @oshun/core-team
Context and Problem Statement#
Isis has RunPod integration at adapter and handler layers, but key production behaviors remain incomplete:
- cost and credit paths are stubbed
- persistent provider job history is not implemented
- typed handler modes are limited versus full Isis workflow catalog
- endpoint lifecycle provisioning/drift control is incomplete
Current code evidence:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.tshas placeholder implementations for model inventory, custom node inventory, cost estimation, credits, and job history.docker/runpod/comfyui/handler.pyvalidates only a limited typed mode set (txt2img,img2img,inpaint,upscale,workflow).infra/terraform/cloudwatch/*includes RunPod monitoring, while endpoint provisioning automation is not defined as desired-state lifecycle resources.
Decision#
Adopt RunPod serverless as the primary asynchronous execution backend for Isis workflow jobs, governed by a capability map, cost ledger, and endpoint desired-state control plane.
Execution Model#
- Use asynchronous RunPod job flow (
run+status+cancel) for production jobs. - Reserve synchronous invocation for explicitly bounded low-latency tooling only.
- Route jobs by
workflowIdvia a versioned capability map to endpoint profiles.
Invocation Contract#
Provider payload must include:
json
{
"workflowId": "string",
"workflowVersion": "string",
"input": {},
"assets": {},
"idempotencyKey": "string",
"trace": {
"jobId": "string",
"projectId": "string",
"userId": "string",
"correlationId": "string"
}
}
Cost and History Governance#
- Persist per-job provider metadata including:
- endpoint id/name
- queued/start/completed timestamps
- runtime seconds
- pricing inputs
- computed
costUsd
- Cost derivation must be deterministic and replayable from stored records.
- Jobs missing billing fields are treated as policy failures.
Endpoint Strategy#
- Define endpoint desired state in code (profile, GPU class, autoscaling limits, timeouts, image version, capability set).
- Validate desired state in CI.
- Run drift detection and reconciliation before releases.
- Use staged rollout/canary by endpoint capability group.
Failure and Fallback Policy#
- RunPod remains primary for serverless execution.
- Secondary provider fallback is allowed only under explicit incident/failover policy and must be observable.
- Unsupported workflow/category responses must be deterministic and machine-readable.
Decision Drivers#
- True pay-per-second production accounting.
- Full workflow catalog coverage with explicit capability routing.
- Operational reliability with drift control and release safety.
- Auditable history for debugging, billing, and compliance.
Considered Options#
Option A: Keep partial RunPod integration and manual ops (Rejected)#
Pros:
- Minimal immediate engineering work.
Cons:
- Incomplete billing/accountability.
- High operational drift risk.
- Weak incident diagnosis and release confidence.
Option B: Switch primary execution to RunComfy and defer RunPod (Rejected)#
Pros:
- Lower immediate RunPod hardening effort.
Cons:
- Conflicts with explicit serverless/pay-per-second objective.
- Defers central remediation findings rather than closing them.
Option C: Harden RunPod as primary async backend with governance controls (Chosen)#
Pros:
- Aligns directly to remediation objective.
- Strong control over cost, capability, and reliability.
- Enables deterministic rollout and operational ownership.
Cons:
- Requires coordinated work across adapter, handler, infra, and observability.
- Requires disciplined desired-state management and CI gates.
Consequences#
Positive#
- RunPod usage becomes auditable, enforceable, and cost-accountable.
- Workflow routing becomes explicit and testable.
- Endpoint drift/regression risk is materially reduced.
Tradeoffs#
- More upfront platform engineering for provisioning and reconciliation tooling.
- More strict validation may initially surface hidden incompatibilities.
Implementation Rules#
- Adapter methods for cost, credits, history, model/custom-node inventory must be non-placeholder in production profiles.
- Handler must expose workflow-id capability contract in addition to legacy typed modes.
- Endpoint capability map is versioned and release-gated.
- Cost and history persistence must be queryable by job/project/user/workflow/endpoint.
- Alerts required for cost spikes, queue growth, failure-rate regression, and missing billing fields.
Rollout Plan#
- Define and publish endpoint capability map and workflow routing policy.
- Replace placeholder adapter billing/history/inventory methods with real integrations.
- Extend handler contracts to catalog-driven workflow execution.
- Implement desired-state endpoint provisioning, drift detection, and reconcile.
- Add canary rollout policy and release gates tied to cost/SLO regressions.
Ticket Traceability#
WS00-005F05-001toF05-010F06-001toF06-010F08-001toF08-010OBS-001toOBS-010REL-001toREL-010
References#
- Internal:
libs/isis/ai-providers/src/providers/comfy/runpod-adapter.tsdocker/runpod/comfyui/handler.pyinfra/terraform/cloudwatch/dashboards.tfinfra/terraform/cloudwatch/alarms.tfdocs/domains/isis/isis-domain-completeness-audit-2026-02-21.md
- External: