Eve library

Alerting Procedures

Guide

Revision 1

On this page

Comprehensive guide for alerting configuration, routing, and response procedures in the Lilith platform.

Table of Contents#

Overview#

The Lilith platform uses a multi-tier alerting system to ensure rapid detection and response to incidents.

Alerting Stack#

Component Technology Purpose
Alert Rules Prometheus/Alertmanager Define alert conditions
Routing Alertmanager Route alerts to channels
Notifications Slack, PagerDuty, Email Deliver alerts
Runbooks Grafana OnCall Response procedures
Dashboards Grafana Alert visualization

Alert Flow#

text
┌─────────────────────────────────────────────────────────────────────────┐
│                          Alert Pipeline                                  │
│                                                                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │ Prometheus  │───▶│ Alertmanager│───▶│  Receivers  │                 │
│  │   Rules     │    │  (Routing)  │    │             │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
│        │                   │                  │                          │
│        ▼                   ▼                  ▼                          │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │  Evaluate   │    │   Group     │    │   Slack     │                 │
│  │ Conditions  │    │   Dedupe    │    │  PagerDuty  │                 │
│  │             │    │   Route     │    │   Email     │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
└─────────────────────────────────────────────────────────────────────────┘

Alert Architecture#

Prometheus Alert Rules#

Alert rules are defined in ConfigMaps and loaded by Prometheus:

yaml
# deploy/terraform/modules/observability/alerting/rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: lilith-alerts
  namespace: monitoring
spec:
  groups:
    - name: availability
      interval: 30s
      rules:
        - alert: ServiceDown
          expr: up{job=~"lilith-.*"} == 0
          for: 1m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: 'Service {{ $labels.job }} is down'
            description:
              'Service {{ $labels.job }} has been down for more than 1 minute.'
            runbook_url: 'https://docs.lilith.app/runbooks/service-down'
            dashboard_url: 'https://grafana.lilith.app/d/service-health'

        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.05
          for: 5m
          labels:
            severity: warning
            team: '{{ $labels.service }}'
          annotations:
            summary: 'High error rate on {{ $labels.service }}'
            description:
              'Error rate is {{ $value | humanizePercentage }} on {{
              $labels.service }}'
            runbook_url: 'https://docs.lilith.app/runbooks/high-error-rate'

Alertmanager Configuration#

yaml
# deploy/terraform/modules/observability/alertmanager/config.yaml
global:
  resolve_timeout: 5m
  slack_api_url: '${SLACK_WEBHOOK_URL}'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Inhibition rules to suppress lower-severity alerts
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'service']

  - source_match:
      alertname: ServiceDown
    target_match_re:
      alertname: .+
    equal: ['service']

# Alert routing tree
route:
  receiver: default-receiver
  group_by: ['alertname', 'severity', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Critical alerts - immediate PagerDuty
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h
      continue: true

    # Warning alerts - Slack
    - match:
        severity: warning
      receiver: slack-warnings
      group_wait: 1m
      repeat_interval: 4h

    # Info alerts - Slack (low priority channel)
    - match:
        severity: info
      receiver: slack-info
      group_wait: 5m
      repeat_interval: 24h

    # Team-specific routing
    - match:
        team: platform
      receiver: platform-team

    - match:
        team: ai
      receiver: ai-team

    - match:
        team: content
      receiver: content-team

receivers:
  - name: default-receiver
    slack_configs:
      - channel: '#alerts-default'
        send_resolved: true

  - name: pagerduty-critical
    pagerduty_configs:
      - service_key: '${PAGERDUTY_SERVICE_KEY}'
        severity: critical
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ template "pagerduty.default.firing" . }}'
          runbook: '{{ .CommonAnnotations.runbook_url }}'

  - name: slack-warnings
    slack_configs:
      - channel: '#alerts-warnings'
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'
        actions:
          - type: button
            text: 'Runbook'
            url: '{{ .CommonAnnotations.runbook_url }}'
          - type: button
            text: 'Dashboard'
            url: '{{ .CommonAnnotations.dashboard_url }}'
          - type: button
            text: 'Silence'
            url: '{{ template "slack.silence_url" . }}'

  - name: slack-info
    slack_configs:
      - channel: '#alerts-info'
        send_resolved: false

  - name: platform-team
    slack_configs:
      - channel: '#platform-alerts'
        send_resolved: true
    pagerduty_configs:
      - service_key: '${PAGERDUTY_PLATFORM_KEY}'

  - name: ai-team
    slack_configs:
      - channel: '#ai-alerts'
        send_resolved: true

  - name: content-team
    slack_configs:
      - channel: '#content-alerts'
        send_resolved: true

Severity Levels#

Severity Definitions#

Severity Response Time Description Examples
Critical < 15 min Service down, data loss, security breach Database unreachable, auth failure
Warning < 1 hour Degraded performance, approaching limits High latency, disk 80%
Info Next business day Informational, non-urgent Deployment complete, backup finished

Severity Matrix#

text
┌────────────────────────────────────────────────────────────────────────┐
│                     Severity Classification Matrix                      │
├────────────────────┬───────────┬───────────┬───────────┬──────────────┤
│ Impact / Urgency   │   High    │  Medium   │    Low    │   Minimal    │
├────────────────────┼───────────┼───────────┼───────────┼──────────────┤
│ High               │ CRITICAL  │ CRITICAL  │  WARNING  │   WARNING    │
│ Medium             │ CRITICAL  │  WARNING  │  WARNING  │    INFO      │
│ Low                │  WARNING  │  WARNING  │   INFO    │    INFO      │
│ Minimal            │  WARNING  │   INFO    │   INFO    │   (none)     │
└────────────────────┴───────────┴───────────┴───────────┴──────────────┘

Severity Labels in Alerts#

yaml
# Critical - Immediate action required
labels:
  severity: critical
  priority: P1
  escalation_minutes: 15

# Warning - Action required within 1 hour
labels:
  severity: warning
  priority: P2
  escalation_minutes: 60

# Info - For awareness
labels:
  severity: info
  priority: P3

Notification Channels#

Slack Configuration#

yaml
# Slack notification template
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# templates/slack.tmpl
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Service:* {{ .Labels.service }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt | date "2006-01-02 15:04:05 MST" }}
{{ if .Annotations.runbook_url }}*Runbook:* {{ .Annotations.runbook_url }}{{ end }}
---
{{ end }}
{{ end }}

PagerDuty Configuration#

yaml
pagerduty_configs:
  - service_key: '${PAGERDUTY_SERVICE_KEY}'
    severity: '{{ .Labels.severity }}'
    client: 'Lilith Alertmanager'
    client_url: 'https://alertmanager.lilith.app'
    description: '{{ .CommonAnnotations.summary }}'
    details:
      firing: '{{ template "pagerduty.default.firing" . }}'
      num_firing: '{{ .Alerts.Firing | len }}'
      num_resolved: '{{ .Alerts.Resolved | len }}'
      resolved: '{{ template "pagerduty.default.resolved" . }}'
    links:
      - href: '{{ .CommonAnnotations.runbook_url }}'
        text: Runbook
      - href: '{{ .CommonAnnotations.dashboard_url }}'
        text: Dashboard
    images:
      - src:
          'https://grafana.lilith.app/render/d-solo/service-{{ .Labels.service
          }}'
        alt: 'Service Dashboard'

Email Configuration#

yaml
email_configs:
  - to: 'oncall@lilith.app'
    from: 'alerts@lilith.app'
    smarthost: 'smtp.ses.amazonaws.com:587'
    auth_username: '${SES_USERNAME}'
    auth_password: '${SES_PASSWORD}'
    send_resolved: true
    headers:
      Subject: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
    html: '{{ template "email.html" . }}'

Alert Routing#

Route Configuration#

yaml
# Detailed routing example
route:
  # Default route
  receiver: default-receiver
  group_by: ['alertname', 'severity', 'service']

  routes:
    # Production critical - page immediately
    - match:
        severity: critical
        environment: production
      receiver: pagerduty-critical
      group_wait: 0s

    # Staging alerts - Slack only
    - match:
        environment: staging
      receiver: slack-staging

    # Database alerts - DBA team
    - match_re:
        alertname: (PostgreSQL|RDS|Database).*
      receiver: dba-team

    # AI service alerts - AI team
    - match:
        service: ai
      receiver: ai-team
      routes:
        - match:
            alertname: GPUOutOfMemory
          receiver: ai-team-gpu

    # Payment alerts - highest priority
    - match:
        service: payment
      receiver: payment-critical
      group_wait: 0s

Silencing Alerts#

bash
# Create a silence via amtool
amtool silence add alertname=ServiceDown service=auth \
  --comment="Planned maintenance" \
  --author="ops@lilith.app" \
  --duration=2h

# List active silences
amtool silence query

# Expire a silence
amtool silence expire <silence-id>

Inhibition Rules#

yaml
inhibit_rules:
  # If cluster is down, suppress service alerts
  - source_match:
      alertname: KubernetesClusterDown
    target_match_re:
      alertname: .+
    equal: ['cluster']

  # If node is down, suppress pod alerts on that node
  - source_match:
      alertname: KubernetesNodeDown
    target_match:
      alertname: KubernetesPodNotReady
    equal: ['node']

  # Suppress lower severity when higher exists
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'service']

Alert Configuration#

Standard Alert Template#

yaml
- alert: AlertName
  expr: <PromQL expression>
  for: <pending duration>
  labels:
    severity: <critical|warning|info>
    team: <team-name>
    service: <service-name>
    environment: '{{ $externalLabels.environment }}'
  annotations:
    summary: 'Short description'
    description: 'Detailed description with {{ $value }}'
    runbook_url: 'https://docs.lilith.app/runbooks/<alert-name>'
    dashboard_url: 'https://grafana.lilith.app/d/<dashboard-id>'

Common Alert Rules#

Infrastructure Alerts#

yaml
groups:
  - name: infrastructure
    rules:
      - alert: NodeHighCPU
        expr:
          100 - (avg by(node) (rate(node_cpu_seconds_total{mode="idle"}[5m])) *
          100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Node {{ $labels.node }} CPU usage is high'
          description: 'CPU usage is {{ $value | printf "%.1f" }}%'

      - alert: NodeHighMemory
        expr:
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) *
          100 > 85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Node {{ $labels.node }} memory usage is high'

      - alert: DiskSpaceLow
        expr:
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
          80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'Disk space low on {{ $labels.node }}'

      - alert: DiskSpaceCritical
        expr:
          (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 >
          95
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Disk space critical on {{ $labels.node }}'

Application Alerts#

yaml
groups:
  - name: application
    rules:
      - alert: HighLatencyP99
        expr:
          histogram_quantile(0.99,
          rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High P99 latency on {{ $labels.service }}'
          description: 'P99 latency is {{ $value | printf "%.2f" }}s'

      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'High error rate on {{ $labels.service }}'

      - alert: ServiceDown
        expr: up{job=~"lilith-.*"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Service {{ $labels.job }} is down'

      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Pod {{ $labels.pod }} is crash looping'

Database Alerts#

yaml
groups:
  - name: database
    rules:
      - alert: PostgreSQLDown
        expr: pg_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'PostgreSQL instance {{ $labels.instance }} is down'

      - alert: PostgreSQLHighConnections
        expr:
          sum(pg_stat_activity_count) by (instance) /
          pg_settings_max_connections > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'PostgreSQL connections at {{ $value | humanizePercentage }}'

      - alert: PostgreSQLSlowQueries
        expr:
          rate(pg_stat_statements_seconds_total[5m]) /
          rate(pg_stat_statements_calls_total[5m]) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Slow queries detected on {{ $labels.instance }}'

      - alert: RedisDown
        expr: redis_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Redis instance {{ $labels.instance }} is down'

Response Procedures#

On-Call Response Flow#

text
┌─────────────────────────────────────────────────────────────────────────┐
│                      Incident Response Flow                              │
│                                                                          │
│  ┌────────────┐    ┌────────────┐    ┌────────────┐    ┌────────────┐  │
│  │   Alert    │───▶│   Triage   │───▶│  Respond   │───▶│  Resolve   │  │
│  │  Received  │    │            │    │            │    │            │  │
│  └────────────┘    └────────────┘    └────────────┘    └────────────┘  │
│        │                 │                  │                 │          │
│        ▼                 ▼                  ▼                 ▼          │
│   Acknowledge       Assess impact      Follow runbook    Document       │
│   in PagerDuty      Check dashboards   Escalate if       Post-mortem    │
│                     Review logs        needed                            │
└─────────────────────────────────────────────────────────────────────────┘

Response Checklist#

markdown
## Critical Alert Response Checklist

### Immediate (0-5 minutes)

- [ ] Acknowledge alert in PagerDuty/Slack
- [ ] Open service dashboard
- [ ] Check for related alerts
- [ ] Identify impacted users/services

### Triage (5-15 minutes)

- [ ] Review recent deployments
- [ ] Check logs for errors
- [ ] Identify root cause or symptoms
- [ ] Communicate status to stakeholders

### Response (15+ minutes)

- [ ] Follow runbook procedures
- [ ] Apply fix or mitigation
- [ ] Verify resolution
- [ ] Clear alert

### Post-Incident

- [ ] Document timeline
- [ ] Create post-mortem if P1/P2
- [ ] Update runbooks if needed

Escalation Matrix#

Level Time Action
L1 0 min Primary on-call paged
L2 15 min Secondary on-call added
L3 30 min Team lead notified
L4 1 hour Engineering manager engaged
L5 2 hours VP Engineering notified

Runbook Integration#

Linking Alerts to Runbooks#

yaml
annotations:
  runbook_url:
    'https://docs.lilith.app/runbooks/{{ $labels.alertname | toLower }}'

Runbook Template#

markdown
# [AlertName] Runbook

## Alert Details

- **Severity**: Critical/Warning/Info
- **Team**: Platform/AI/Content
- **Service**: service-name

## Description

What this alert means and why it fires.

## Impact

- User-facing impact
- Business impact
- Data impact

## Investigation Steps

1. Check the dashboard: [link]
2. Review logs: `kubectl logs -n production -l app=service-name`
3. Check recent deployments
4. Review metrics

## Resolution Steps

1. Step-by-step fix
2. Verification steps
3. Rollback procedure if needed

## Prevention

- How to prevent recurrence
- Recommended monitoring improvements

Best Practices#

Alert Design#

  1. Be specific: Alert on symptoms, not causes
  2. Include context: Add relevant labels and annotations
  3. Set appropriate thresholds: Avoid alert fatigue
  4. Use pending duration: Prevent flapping alerts
  5. Link runbooks: Every alert should have a runbook

Alert Hygiene#

yaml
# Good: Specific, actionable
- alert: AuthServiceHighLatencyP99
  expr:
    histogram_quantile(0.99,
    rate(http_request_duration_seconds_bucket{service="auth"}[5m])) > 0.5
  for: 5m
  annotations:
    summary: 'Auth service P99 latency exceeds 500ms'
    runbook_url: '...'

# Bad: Too generic, no context
- alert: HighLatency
  expr:
    histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1

Reducing Alert Fatigue#

  1. Group related alerts: Use group_by effectively
  2. Use inhibition: Suppress redundant alerts
  3. Set repeat intervals: Don't spam the same alert
  4. Review regularly: Remove or adjust noisy alerts

Testing Alerts#

bash
# Test alert rule syntax
promtool check rules rules.yaml

# Test alertmanager config
amtool check-config alertmanager.yml

# Send test alert
curl -X POST http://alertmanager:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning"
    },
    "annotations": {
      "summary": "This is a test alert"
    }
  }]'

Troubleshooting#

Common Issues#

Issue Cause Solution
Alert not firing Expression wrong Test in Prometheus UI
Alert not routing Route mismatch Check route labels
No notifications Receiver config wrong Test receiver endpoint
Too many alerts Thresholds too low Adjust thresholds

Debug Commands#

bash
# Check Alertmanager status
curl http://alertmanager:9093/api/v1/status

# View active alerts
curl http://alertmanager:9093/api/v1/alerts

# Check routing
amtool config routes show --config.file=alertmanager.yml

# Test routing for specific labels
amtool config routes test --config.file=alertmanager.yml \
  severity=critical service=auth