Eve library

GATE-007 Certification: Observability Dashboards and Alerts Validated

Guide

Revision 1

On this page

Date: 2026-02-22
Checklist Item: GATE-007
Ticket: ISIS-0257

Scope#

Certify that Isis observability surfaces for lifecycle, provider reliability, queue health, and cost anomaly detection are defined, wired, and validated via deterministic checks.

Evidence Matrix#

Requirement Evidence Status
Isis provider operations dashboard is defined for latency/failure/volume infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_provider_operations includes ProviderExecutionLatencyMs, ProviderExecutionFailed, and ProviderExecutionTotal visualizations. Pass
Isis cost dashboard is defined across endpoint/project/workflow/user dimensions infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_cost includes ProviderExecutionCostByEndpointUsd, ProviderExecutionCostByProjectUsd, ProviderExecutionCostByWorkflowUsd, and ProviderExecutionCostByUserUsd. Pass
Isis generation lifecycle dashboard is defined infra/terraform/cloudwatch/dashboards.tf resource aws_cloudwatch_dashboard.isis_generation includes lifecycle metrics GenerationStarted, GenerationCompleted, and GenerationFailed. Pass
Provider anomaly alerts are defined for cost and failure-rate spikes infra/terraform/cloudwatch/alarms.tf resources aws_cloudwatch_metric_alarm.isis_provider_cost_spike_anomaly and aws_cloudwatch_metric_alarm.isis_provider_failure_rate_spike_anomaly use ANOMALY_DETECTION_BAND against provider execution cost/failure metrics. Pass
Queue/failure alerting path is defined infra/terraform/cloudwatch/alarms.tf includes aws_cloudwatch_metric_alarm.runpod_queue_depth and aws_cloudwatch_metric_alarm.runpod_high_failure_rate. Pass
Metric ingestion for Isis provider telemetry is wired from canonical service logs infra/terraform/cloudwatch/logs.tf scopes provider metric filters to Isis generation-api (v.domain == "isis" && v.service == "generation-api") and emits ProviderExecution* + ProviderExecutionCost* metrics used by dashboards/alarms. Pass
Dashboard/alarm outputs are exported for operational consumption infra/terraform/cloudwatch/outputs.tf exports isis_provider_ops, isis_cost, isis_business, and provider_anomaly_alarm_arns. Pass
SLO metric contract remains aligned with observability gate docs/domains/isis/remediation/ISIS_SLO_DEFINITIONS.md defines canonical metrics (isis_generation_submit_latency_ms, isis_generation_queue_delay_seconds, isis_generation_terminal_total, isis_generation_e2e_completion_seconds). Pass
Automated validation gate exists and passes scripts/isis/validate_observability_stack.mjs validates dashboards, alarms, metric filters, outputs, and SLO metric contracts; command succeeded with Isis observability stack validation passed (67 checks). Pass
Alert dispatch behavior for readiness degradation remains tested apps/isis/generation-api/src/services/readiness-alert.service.spec.ts passed to verify readiness alert dispatch + cooldown behavior for operational alert signal flow. Pass
On-call runbook references the certified dashboard/alarm set docs/domains/isis/runbooks/runpod-oncall.md lists primary CloudWatch dashboards and anomaly alert triggers used for incident response and operational validation. Pass

Reproduction Commands#

bash
node scripts/isis/validate_observability_stack.mjs
pnpm --filter @isis/generation-api exec vitest run src/services/readiness-alert.service.spec.ts