Eve library

Troubleshooting Guide (Rapid Triage)

Runbook

Revision 1

On this page

Use this playbook during incidents to quickly isolate and remediate issues across Lilith services.

1) Classify the Failure#

  • Availability: 5xx spikes, timeouts, failing health checks.
  • Degradation: Elevated p95 latency, partial feature failures, slow queues.
  • Data Integrity: Migrations, data loss/corruption indicators.
  • Security: Suspected breach, secrets exposure, auth anomalies.

2) Immediate Checks (5 minutes)#

  • Verify health endpoints:
    bash
    for s in bff auth content ai; do curl -sf https://$s.lilith.app/health || echo "$s unhealthy"; done
    
  • Inspect service logs for the timeframe:
    bash
    kubectl logs deploy/bff-production -n production --since=10m | tail -n 200
    
  • Check error ratios and latency:
    bash
    ./scripts/observability/query-slo.sh --window 10m
    
  • Confirm dependency health (DB/Redis/queues).

3) Contain#

  • Flip feature flags for risky paths.
  • Scale up replicas if saturation is visible:
    bash
    kubectl scale deploy/bff-production -n production --replicas=+2
    
  • If release-related, route traffic back to stable (see rollback runbook).

4) Diagnose by Layer#

  • Network/Ingress: Cert expiry, DNS changes, rate limiters, WAF blocks.
  • App: Recent deployments, uncaught exceptions, config/env drift.
  • Data: Slow queries, lock contention, migration effects, cache stampedes.
  • External: Third-party dependency status pages and timeouts.

5) Remediate#

  • Apply hotfix or rollback using the deployment rollback runbook.
  • Clear poison messages / replay queues if stuck consumers.
  • Rebuild caches behind feature flags to avoid thundering herd.
  • For data issues, switch to read-only mode and engage DB on-call.

6) Communicate#

  • Post status in #api-alerts with impact, scope, and ETA.
  • Update the incident doc with timeline and actions.
  • If customer impact is confirmed, page comms owner for outbound updates.

7) Verify and Close#

  • Confirm SLOs recovered (error rate, latency).
  • Run smoke tests:
    bash
    npm run test:integration -- --grep "smoke"
    
  • Capture PIR with root cause, detection gap, and regression tests added.
  • File follow-up tickets for infra/app fixes and assign owners.