This document provides runbooks for Kubernetes and infrastructure-related alerts in the Lilith platform.
Overview#
Infrastructure alerts are triggered by the kube-prometheus-stack and monitor the health of Kubernetes clusters, nodes, pods, and core infrastructure components.
Alert Index#
| Alert | Severity | Description |
|---|---|---|
| KubePodCrashLooping | Critical | Pod restarting repeatedly |
| KubePodNotReady | Warning | Pod not in ready state |
| KubeDeploymentReplicasMismatch | Warning | Deployment has wrong replica count |
| KubeNodeNotReady | Critical | Node not in ready state |
| KubeNodeUnreachable | Critical | Node unreachable |
| KubeCPUOvercommit | Warning | CPU overcommitted on cluster |
| KubeMemoryOvercommit | Warning | Memory overcommitted on cluster |
| NodeDiskRunningFull | Critical | Disk space running low |
| NodeHighCPUUsage | Warning | High CPU utilization |
| NodeHighMemoryUsage | Warning | High memory utilization |
KubePodCrashLooping#
Description#
A pod is restarting repeatedly (crash loop). This typically indicates an application issue, misconfiguration, or resource constraint.
Severity#
Critical - Immediate attention required
Trigger Condition#
rate(kube_pod_container_status_restarts_total{namespace=~"lilith.*"}[15m]) * 60 * 5 > 0
Impact#
- Service degradation or outage
- Potential data inconsistency
- User-facing errors
Investigation Steps#
-
Identify the crashing pod
bashkubectl get pods -n <namespace> | grep -E "(CrashLoopBackOff|Error)" -
Check pod events
bashkubectl describe pod <pod-name> -n <namespace> -
Check logs from previous container
bashkubectl logs <pod-name> -n <namespace> --previous kubectl logs <pod-name> -n <namespace> --tail=100 -
Check resource usage
bashkubectl top pod <pod-name> -n <namespace> -
Check for OOMKilled
bashkubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
Remediation Actions#
-
If OOMKilled:
bash# Increase memory limits kubectl patch deployment <deployment> -n <namespace> \ -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"limits":{"memory":"1Gi"}}}]}}}}' -
If configuration error:
- Check ConfigMaps and Secrets
- Verify environment variables
- Review startup command
-
If dependency unavailable:
- Check dependent services
- Verify database connectivity
- Check network policies
-
If code bug:
- Roll back to last known good version
- Escalate to development team
Escalation Path#
- On-call SRE
- Service team (if application-related)
- Platform team (if infrastructure-related)
KubePodNotReady#
Description#
A pod is running but not in the Ready state. This means the readiness probe is failing.
Severity#
Warning - Investigate within 15 minutes
Trigger Condition#
kube_pod_status_ready{condition="false", namespace=~"lilith.*"} == 1
Investigation Steps#
-
Check pod status
bashkubectl get pods -n <namespace> -o wide kubectl describe pod <pod-name> -n <namespace> -
Check readiness probe configuration
bashkubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].readinessProbe}' -
Test readiness endpoint manually
bashkubectl exec -it <pod-name> -n <namespace> -- curl localhost:<port>/health/ready -
Check logs for errors
bashkubectl logs <pod-name> -n <namespace> --tail=50
Remediation Actions#
-
If probe failing due to dependency:
- Check dependent services
- Consider adjusting probe settings
-
If probe misconfigured:
- Review probe path and port
- Adjust initialDelaySeconds if startup is slow
-
If resource-related:
- Check CPU/memory utilization
- Scale resources if needed
KubeDeploymentReplicasMismatch#
Description#
The deployment has fewer replicas than desired. Pods may be pending, failed, or unable to schedule.
Severity#
Warning - Investigate within 30 minutes
Trigger Condition#
kube_deployment_status_replicas_available != kube_deployment_spec_replicas
Investigation Steps#
-
Check deployment status
bashkubectl get deployment <deployment> -n <namespace> kubectl describe deployment <deployment> -n <namespace> -
Check for pending pods
bashkubectl get pods -n <namespace> | grep Pending kubectl describe pod <pending-pod> -n <namespace> -
Check for scheduling issues
bashkubectl get events -n <namespace> --field-selector reason=FailedScheduling -
Check node resources
bashkubectl describe nodes | grep -A5 "Allocated resources"
Remediation Actions#
-
If scheduling failure due to resources:
bash# Scale up node group # Or reduce resource requests kubectl patch deployment <deployment> -n <namespace> \ -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","resources":{"requests":{"memory":"256Mi","cpu":"100m"}}}]}}}}' -
If node affinity issue:
- Review node affinity rules
- Add appropriate node labels
- Consider removing constraints
-
If PVC issues:
- Check PersistentVolumeClaim status
- Verify storage class availability
KubeNodeNotReady#
Description#
A Kubernetes node is not in the Ready state. Pods on this node may be affected.
Severity#
Critical - Immediate attention required
Trigger Condition#
kube_node_status_condition{condition="Ready",status="true"} == 0
Investigation Steps#
-
Check node status
bashkubectl get nodes kubectl describe node <node-name> -
Check node conditions
bashkubectl get node <node-name> -o jsonpath='{.status.conditions[*]}' -
Check kubelet status (if accessible)
bashssh <node> systemctl status kubelet ssh <node> journalctl -u kubelet --since="10 minutes ago" -
Check for resource pressure
bashkubectl describe node <node-name> | grep -A10 "Conditions:"
Remediation Actions#
-
If node is recoverable:
bash# Restart kubelet ssh <node> sudo systemctl restart kubelet -
If node is unrecoverable:
bash# Cordon and drain kubectl cordon <node-name> kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data # Replace node (EKS/GKE/AKS specific) -
If disk pressure:
- Clean up unused images
- Expand disk or clean logs
-
If memory pressure:
- Evict low-priority pods
- Add capacity to cluster
Escalation Path#
- On-call SRE
- Platform team
- Cloud provider support (if hardware issue)
KubeNodeUnreachable#
Description#
A Kubernetes node cannot be reached by the control plane.
Severity#
Critical - Immediate attention required
Investigation Steps#
-
Check node status
bashkubectl get nodes -
Check cloud provider console
- Verify instance is running
- Check network connectivity
- Review security groups
-
Check VPC and networking
- Verify VPC routes
- Check NAT gateway status
- Review network ACLs
Remediation Actions#
-
If network issue:
- Check security group rules
- Verify VPC configuration
- Review network policies
-
If instance crashed:
- Attempt instance restart via cloud console
- If unrecoverable, terminate and replace
KubeCPUOvercommit#
Description#
The cluster has overcommitted CPU - sum of CPU requests exceeds available capacity.
Severity#
Warning - Plan capacity increase
Trigger Condition#
sum(kube_pod_container_resource_requests{resource="cpu"})
/
sum(kube_node_status_allocatable{resource="cpu"}) > 1
Investigation Steps#
-
Check cluster CPU allocation
bashkubectl describe nodes | grep -A10 "Allocated resources" -
Find highest CPU consumers
bashkubectl top pods --all-namespaces --sort-by=cpu | head -20
Remediation Actions#
-
Add cluster capacity:
- Scale up node group
- Add new node pool
-
Optimize requests:
- Review pod CPU requests
- Right-size based on actual usage
KubeMemoryOvercommit#
Description#
The cluster has overcommitted memory - sum of memory requests exceeds available capacity.
Severity#
Warning - Plan capacity increase
Investigation Steps#
-
Check cluster memory allocation
bashkubectl describe nodes | grep -A10 "Allocated resources" -
Find highest memory consumers
bashkubectl top pods --all-namespaces --sort-by=memory | head -20
Remediation Actions#
Same as CPU overcommit - add capacity or optimize requests.
NodeDiskRunningFull#
Description#
Node disk space is running low (< 15% free).
Severity#
Critical - Immediate attention required
Trigger Condition#
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
Investigation Steps#
-
Check disk usage
bashdf -h du -sh /* | sort -hr | head -20 -
Check for large log files
bashfind /var/log -type f -size +100M -
Check container images
bashdocker system df crictl images | wc -l
Remediation Actions#
-
Clean up Docker/containerd images
bashdocker system prune -a crictl rmi --prune -
Clean up old logs
bashjournalctl --vacuum-time=7d find /var/log -type f -name "*.gz" -mtime +7 -delete -
Expand disk (cloud-specific)
- EKS: Modify EBS volume, then resize filesystem
- GKE: Edit node pool disk size
- AKS: Scale VM disk
NodeHighCPUUsage#
Description#
Node CPU utilization is consistently high (> 90%).
Severity#
Warning - Investigate and plan capacity
Trigger Condition#
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
Investigation Steps#
-
Identify high CPU processes
bashkubectl top pods -A --sort-by=cpu | head -20 -
Check for CPU throttling
promqlrate(container_cpu_cfs_throttled_seconds_total[5m])
Remediation Actions#
- Scale out workloads to distribute load
- Increase node size if consistently overloaded
- Optimize application if specific service is the cause
NodeHighMemoryUsage#
Description#
Node memory utilization is high (> 90%).
Severity#
Warning - Risk of OOM conditions
Trigger Condition#
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
Investigation Steps#
-
Identify high memory pods
bashkubectl top pods -A --sort-by=memory | head -20 -
Check for memory leaks
promqlcontainer_memory_usage_bytes{namespace=~"lilith.*"}[1h]
Remediation Actions#
- Scale or restart memory-leaking pods
- Increase node size
- Add memory limits to prevent runaway processes
General Commands#
# Cluster overview
kubectl get nodes -o wide
kubectl top nodes
# Pod status across namespaces
kubectl get pods -A | grep -v Running
# Recent events
kubectl get events -A --sort-by='.lastTimestamp' | tail -30
# Resource allocation
kubectl describe nodes | grep -A20 "Allocated resources"
Contact Information#
| Role | Contact |
|---|---|
| On-Call SRE | #sre-oncall (Slack) |
| Platform Team | #platform-team (Slack) |
| Cloud Team | #cloud-infrastructure (Slack) |