Date: 2026-02-22
Checklist Item: GATE-007
Ticket: ISIS-0257
Scope#
Certify that Isis observability surfaces for lifecycle, provider reliability, queue health, and cost anomaly detection are defined, wired, and validated via deterministic checks.
Evidence Matrix#
| Requirement | Evidence | Status |
|---|---|---|
| Isis provider operations dashboard is defined for latency/failure/volume | infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_provider_operations includes ProviderExecutionLatencyMs, ProviderExecutionFailed, and ProviderExecutionTotal visualizations. |
Pass |
| Isis cost dashboard is defined across endpoint/project/workflow/user dimensions | infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_cost includes ProviderExecutionCostByEndpointUsd, ProviderExecutionCostByProjectUsd, ProviderExecutionCostByWorkflowUsd, and ProviderExecutionCostByUserUsd. |
Pass |
| Isis generation lifecycle dashboard is defined | infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_generation includes lifecycle metrics GenerationStarted, GenerationCompleted, and GenerationFailed. |
Pass |
| Provider anomaly alerts are defined for cost and failure-rate spikes | infra/terraform/cloudwatch/alarms.tf resources aws_cloudwatch_metric_alarm.isis_provider_cost_spike_anomaly and aws_cloudwatch_metric_alarm.isis_provider_failure_rate_spike_anomaly use ANOMALY_DETECTION_BAND against provider execution cost/failure metrics. |
Pass |
| Queue/failure alerting path is defined | infra/terraform/cloudwatch/alarms.tf includes aws_cloudwatch_metric_alarm.runpod_queue_depth and aws_cloudwatch_metric_alarm.runpod_high_failure_rate. |
Pass |
| Metric ingestion for Isis provider telemetry is wired from canonical service logs | infra/terraform/cloudwatch/logs.tf scopes provider metric filters to Isis generation-api (v.domain == "isis" && v.service == "generation-api") and emits ProviderExecution* + ProviderExecutionCost* metrics used by dashboards/alarms. |
Pass |
| Dashboard/alarm outputs are exported for operational consumption | infra/terraform/cloudwatch/outputs.tf exports isis_provider_ops, isis_cost, isis_business, and provider_anomaly_alarm_arns. |
Pass |
| SLO metric contract remains aligned with observability gate | docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md defines canonical metrics (isis_generation_submit_latency_ms, isis_generation_queue_delay_seconds, isis_generation_terminal_total, isis_generation_e2e_completion_seconds). |
Pass |
| Automated validation gate exists and passes | scripts/isis/validate_observability_stack.mjs validates dashboards, alarms, metric filters, outputs, and SLO metric contracts; command succeeded with Isis observability stack validation passed (67 checks). |
Pass |
| Alert dispatch behavior for readiness degradation remains tested | apps/isis/generation-api/src/services/readiness-alert.service.spec.ts passed to verify readiness alert dispatch + cooldown behavior for operational alert signal flow. |
Pass |
| On-call runbook references the certified dashboard/alarm set | docs/domains/isis/runbooks/runpod-oncall.md lists primary CloudWatch dashboards and anomaly alert triggers used for incident response and operational validation. |
Pass |
Reproduction Commands#
bash
node scripts/isis/validate_observability_stack.mjs
pnpm --filter @isis/generation-api exec vitest run src/services/readiness-alert.service.spec.ts