# Application Alert Runbooks

This document provides runbooks for application-specific alerts in the Lilith
platform.

## Overview

Application alerts monitor the health and performance of Lilith services
including auth, content, conversation, media, notification, and AI services.

## Alert Index

| Alert                                                       | Severity         | Description                         |
| ----------------------------------------------------------- | ---------------- | ----------------------------------- |
| [ServiceDown](#servicedown)                                 | Critical         | Service not responding              |
| [HighErrorRate](#higherrorrate)                             | Critical/Warning | High percentage of failed requests  |
| [HighLatency](#highlatency)                                 | Warning          | Response times exceeding thresholds |
| [DatabaseConnectionFailure](#databaseconnectionfailure)     | Critical         | Cannot connect to database          |
| [QueueBacklog](#queuebacklog)                               | Warning          | Message queue backlog growing       |
| [RateLimitExceeded](#ratelimitexceeded)                     | Warning          | Rate limiting being triggered       |
| [AuthenticationFailures](#authenticationfailures)           | Warning          | Unusual authentication failures     |
| [AIServiceDegraded](#aiservicedegraded)                     | Warning          | AI service performance degraded     |
| [MediaProcessingBacklog](#mediaprocessingbacklog)           | Warning          | Media processing queue growing      |
| [NotificationDeliveryFailure](#notificationdeliveryfailure) | Warning          | Notifications failing to deliver    |

---

## ServiceDown

### Description

A service is not responding to health checks and is considered down.

### Severity

**Critical** - Immediate attention required

### Trigger Condition

```promql
up{job=~"lilith-.*"} == 0
```

### Impact

- Complete service outage
- User-facing errors
- Potential data loss for in-flight requests

### Investigation Steps

1. **Check pod status**

   ```bash
   kubectl get pods -n lilith -l app=<service>
   kubectl describe pods -n lilith -l app=<service>
   ```

2. **Check recent events**

   ```bash
   kubectl get events -n lilith --sort-by='.lastTimestamp' | grep <service>
   ```

3. **Check logs**

   ```bash
   kubectl logs -n lilith -l app=<service> --tail=100
   kubectl logs -n lilith -l app=<service> --previous --tail=100
   ```

4. **Check health endpoint**

   ```bash
   kubectl exec -it <any-pod> -n lilith -- curl http://<service>:8080/health
   ```

5. **Check dependencies**
   - Database connectivity
   - Cache availability
   - External service access

### Remediation Actions

1. **If pods are crashing:**

   ```bash
   # Check crash reason and follow KubePodCrashLooping runbook
   kubectl logs <pod> -n lilith --previous
   ```

2. **If pods are pending:**

   ```bash
   # Check scheduling issues
   kubectl describe pod <pod> -n lilith | grep -A20 Events
   ```

3. **If networking issue:**

   ```bash
   # Check service endpoints
   kubectl get endpoints <service> -n lilith
   kubectl get svc <service> -n lilith
   ```

4. **Restart deployment:**
   ```bash
   kubectl rollout restart deployment/<service> -n lilith
   kubectl rollout status deployment/<service> -n lilith
   ```

### Escalation Path

1. On-call SRE
2. Service team lead
3. Platform team

---

## HighErrorRate

### Description

The service is returning a high percentage of error responses (5xx status
codes).

### Severity

- **Critical** - Error rate > 10%
- **Warning** - Error rate > 1%

### Trigger Condition

```promql
sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m]))
/
sum(rate(http_requests_total{service="<service>"}[5m])) > 0.01
```

### Investigation Steps

1. **Identify error types**

   ```promql
   sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (code)
   ```

2. **Find affected endpoints**

   ```promql
   topk(10, sum(rate(http_requests_total{code=~"5..", service="<service>"}[5m])) by (handler))
   ```

3. **Check logs for errors**

   ```bash
   kubectl logs -n lilith -l app=<service> --tail=500 | grep -i "error\|exception\|fatal"
   ```

4. **Check traces for failures**
   - Open Tempo/Jaeger
   - Filter for error spans
   - Identify root cause span

5. **Check dependency health**
   ```bash
   kubectl get pods -n lilith
   # Check database, cache, external services
   ```

### Remediation Actions

1. **If dependency failure:**
   - Address dependency issue first
   - Enable circuit breaker if available
   - Consider graceful degradation

2. **If code bug:**

   ```bash
   # Rollback to last known good version
   kubectl rollout undo deployment/<service> -n lilith
   ```

3. **If resource exhaustion:**

   ```bash
   # Scale up
   kubectl scale deployment/<service> -n lilith --replicas=<n+2>
   ```

4. **If rate limit related:**
   - Increase rate limits temporarily
   - Add caching to reduce load

---

## HighLatency

### Description

Service response times are exceeding acceptable thresholds.

### Severity

**Warning** - Investigate within 30 minutes

### Trigger Condition

```promql
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le)) > 1
```

### Investigation Steps

1. **Identify slow endpoints**

   ```promql
   topk(10, histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="<service>"}[5m])) by (le, handler)))
   ```

2. **Check database performance**

   ```promql
   histogram_quantile(0.95, sum(rate(db_query_duration_seconds_bucket{service="<service>"}[5m])) by (le, query))
   ```

3. **Check external call latency**

   ```promql
   histogram_quantile(0.95, sum(rate(external_call_duration_seconds_bucket{service="<service>"}[5m])) by (le, target))
   ```

4. **Review traces**
   - Open Tempo/Jaeger
   - Filter for slow traces
   - Identify slowest span

5. **Check resource usage**
   ```bash
   kubectl top pods -n lilith -l app=<service>
   ```

### Remediation Actions

1. **If database slow:**
   - Check for missing indexes
   - Review query plans
   - Add read replica

2. **If resource constrained:**

   ```bash
   kubectl scale deployment/<service> -n lilith --replicas=<n+2>
   ```

3. **If external dependency slow:**
   - Add caching
   - Increase timeout
   - Consider async processing

4. **If N+1 queries:**
   - Review code for batch loading
   - Add eager loading

---

## DatabaseConnectionFailure

### Description

Service cannot establish connections to the database.

### Severity

**Critical** - Immediate attention required

### Trigger Condition

```promql
sum(db_connection_errors_total{service="<service>"}) > 0
```

### Investigation Steps

1. **Check database status**

   ```bash
   # Check RDS console or
   aws rds describe-db-instances --db-instance-identifier <instance-id>
   ```

2. **Check connection pool**

   ```promql
   db_pool_connections_active{service="<service>"}
   db_pool_connections_idle{service="<service>"}
   ```

3. **Check network connectivity**

   ```bash
   kubectl exec -it <pod> -n lilith -- nc -zv <db-host> 5432
   ```

4. **Check credentials**
   ```bash
   kubectl get secret <db-secret> -n lilith -o yaml
   ```

### Remediation Actions

1. **If database down:**
   - Check RDS events in AWS console
   - Initiate failover if needed

2. **If connection exhaustion:**
   - Increase connection pool size
   - Check for connection leaks
   - Scale down pods to reduce pressure

3. **If network issue:**
   - Check security groups
   - Verify VPC peering
   - Check DNS resolution

---

## QueueBacklog

### Description

Message queue (SQS, RabbitMQ, etc.) has a growing backlog of unprocessed
messages.

### Severity

**Warning** - Investigate within 30 minutes

### Trigger Condition

```promql
queue_messages_pending{queue="<queue-name>"} > 1000
```

### Investigation Steps

1. **Check queue metrics**

   ```promql
   queue_messages_pending{queue="<queue-name>"}
   rate(queue_messages_processed_total{queue="<queue-name>"}[5m])
   ```

2. **Check consumer health**

   ```bash
   kubectl get pods -n lilith -l app=<consumer-service>
   kubectl logs -n lilith -l app=<consumer-service> --tail=100
   ```

3. **Check for processing errors**
   ```promql
   rate(queue_messages_failed_total{queue="<queue-name>"}[5m])
   ```

### Remediation Actions

1. **Scale up consumers:**

   ```bash
   kubectl scale deployment/<consumer> -n lilith --replicas=<n+2>
   ```

2. **If consumer failing:**
   - Fix consumer issue first
   - Consider DLQ review

3. **If producer spike:**
   - Verify traffic patterns
   - Consider rate limiting producers

---

## RateLimitExceeded

### Description

API rate limits are being triggered, indicating high traffic or potential abuse.

### Severity

**Warning** - Monitor and investigate

### Trigger Condition

```promql
sum(rate(http_requests_total{code="429"}[5m])) > 10
```

### Investigation Steps

1. **Identify affected clients**

   ```promql
   topk(10, sum(rate(http_requests_total{code="429"}[5m])) by (client_id))
   ```

2. **Check traffic patterns**

   ```promql
   sum(rate(http_requests_total{service="<service>"}[5m])) by (handler)
   ```

3. **Review for abuse patterns**
   - Check client IPs
   - Review request patterns
   - Look for bot signatures

### Remediation Actions

1. **If legitimate traffic:**
   - Increase rate limits
   - Add caching
   - Scale infrastructure

2. **If abuse:**
   - Block offending IPs/clients
   - Add additional security measures
   - Report to security team

---

## AuthenticationFailures

### Description

Unusual number of authentication failures detected.

### Severity

**Warning** - May indicate attack or misconfiguration

### Trigger Condition

```promql
sum(rate(lilith_auth_failures_total[5m])) > 100
```

### Investigation Steps

1. **Check failure types**

   ```promql
   sum(rate(lilith_auth_failures_total[5m])) by (reason)
   ```

2. **Identify sources**

   ```promql
   topk(10, sum(rate(lilith_auth_failures_total[5m])) by (client_ip))
   ```

3. **Check for patterns**
   - Same user, multiple attempts
   - Same IP, different users
   - Geographic anomalies

### Remediation Actions

1. **If brute force attack:**
   - Block offending IPs
   - Enable additional rate limiting
   - Alert security team

2. **If misconfiguration:**
   - Check OAuth/JWT settings
   - Verify secret rotation
   - Review recent changes

---

## AIServiceDegraded

### Description

AI service (LLM, embeddings, etc.) is experiencing degraded performance.

### Severity

**Warning** - User experience impact

### Trigger Condition

```promql
histogram_quantile(0.95, sum(rate(lilith_ai_request_duration_seconds_bucket[5m])) by (le)) > 30
```

### Investigation Steps

1. **Check AI provider status**
   - OpenAI/Anthropic status pages
   - Check for announced outages

2. **Check queue depth**

   ```promql
   lilith_ai_queue_depth
   ```

3. **Check token usage**
   ```promql
   sum(rate(lilith_ai_tokens_used_total[5m])) by (model)
   ```

### Remediation Actions

1. **If provider issue:**
   - Switch to backup provider
   - Enable fallback responses
   - Communicate to users

2. **If rate limited:**
   - Reduce request rate
   - Enable request queuing
   - Consider upgrading plan

---

## MediaProcessingBacklog

### Description

Media processing queue has accumulated a backlog.

### Severity

**Warning** - Delayed media availability

### Investigation Steps

1. **Check queue status**

   ```promql
   lilith_media_processing_queue_depth
   ```

2. **Check processing times**

   ```promql
   histogram_quantile(0.95, sum(rate(lilith_media_processing_duration_seconds_bucket[5m])) by (le, type))
   ```

3. **Check worker health**
   ```bash
   kubectl get pods -n lilith -l app=media-processor
   ```

### Remediation Actions

1. **Scale up processors:**

   ```bash
   kubectl scale deployment/media-processor -n lilith --replicas=<n+2>
   ```

2. **If GPU constrained:**
   - Add GPU nodes
   - Prioritize queue

---

## NotificationDeliveryFailure

### Description

Notifications (push, email, SMS) are failing to deliver.

### Severity

**Warning** - User communication impacted

### Trigger Condition

```promql
sum(rate(lilith_notifications_failed_total[5m])) / sum(rate(lilith_notifications_sent_total[5m])) > 0.05
```

### Investigation Steps

1. **Check failure reasons**

   ```promql
   sum(rate(lilith_notifications_failed_total[5m])) by (channel, reason)
   ```

2. **Check provider status**
   - FCM/APNs status
   - Email provider status
   - SMS provider status

### Remediation Actions

1. **If push token issues:**
   - Clean up stale tokens
   - Check app registration

2. **If email issues:**
   - Check SPF/DKIM
   - Review bounce rates
   - Check sending limits

---

## General Troubleshooting

### Useful PromQL Queries

```promql
# Error rate by service
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Latency by service
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Request rate by service
sum(rate(http_requests_total[5m])) by (service)
```

### Quick Diagnostics

```bash
# Service health check
for svc in auth bff content conversation media notification ai; do
  echo "Checking $svc..."
  kubectl get pods -n lilith -l app=$svc -o wide
done

# Recent errors in all services
for svc in auth bff content conversation media notification ai; do
  echo "=== $svc errors ==="
  kubectl logs -n lilith -l app=$svc --since=5m | grep -i error | tail -5
done
```

## Contact Information

| Team          | Contact        | Escalation                 |
| ------------- | -------------- | -------------------------- |
| Auth Team     | #auth-team     | auth-oncall@lilith.app     |
| Content Team  | #content-team  | content-oncall@lilith.app  |
| AI Team       | #ai-team       | ai-oncall@lilith.app       |
| Platform Team | #platform-team | platform-oncall@lilith.app |

## References

- [Service Architecture](../../ARCHITECTURE.md)
- [API Documentation](../../api-specs/)
- [Incident Response](../incidents/incident-response.md)
