# Monitoring and Observability

## Overview

Oshun uses a layered observability strategy: OpenTelemetry for instrumentation,
Prometheus and Grafana for metrics and dashboards, Jaeger for distributed
tracing, and CloudWatch for production AWS monitoring.

---

## Development Environment

### Starting Observability Stack

The observability services are available as a Docker Compose profile:

```bash
docker compose -f docker/docker-compose.dev.yml --profile observability up -d
```

This starts:

| Service        | Port  | Purpose                    | Credentials |
| -------------- | ----- | -------------------------- | ----------- |
| **Prometheus** | 9090  | Metrics scraping & storage | None        |
| **Grafana**    | 3100  | Dashboards & visualization | admin/admin |
| **Jaeger**     | 16686 | Distributed tracing UI     | None        |

### Prometheus

**URL:** http://localhost:9090

Configuration file: `docker/observability/prometheus.dev.yml`

Prometheus scrapes metrics from all services exposing a `/metrics` endpoint.
Services use `@oshun/metrics` (which wraps `prom-client`) to expose Prometheus
metrics.

### Grafana

**URL:** http://localhost:3100

Default credentials: `admin` / `admin`

Grafana connects to Prometheus as a data source and provides pre-built
dashboards. Dashboard provisioning configs are in
`docker/observability/grafana/provisioning/`.

### Jaeger

**URL:** http://localhost:16686

Jaeger collects distributed traces via OpenTelemetry. Services use
`@oshun/tracing` to instrument HTTP requests, database queries, and
inter-service calls.

**Ports:**

- `16686` - UI
- `14268` - Collector (HTTP)
- `6831/udp` - Agent (Thrift compact)
- `9411` - Zipkin-compatible endpoint

---

## Instrumentation Libraries

### `@oshun/metrics`

Wraps `prom-client` to provide:

- HTTP request duration histograms
- Request count counters
- Active connection gauges
- Custom domain-specific metrics
- `/metrics` endpoint middleware

### `@oshun/tracing`

Wraps OpenTelemetry SDK to provide:

- Automatic HTTP request tracing
- Database query instrumentation
- Inter-service propagation (W3C Trace Context)
- Custom span creation

**OpenTelemetry packages used:**

- `@opentelemetry/api`
- `@opentelemetry/sdk-trace-node`
- `@opentelemetry/sdk-trace-base`
- `@opentelemetry/exporter-trace-otlp-http`
- `@opentelemetry/resources`
- `@opentelemetry/semantic-conventions`
- `@opentelemetry/context-async-hooks`
- `@opentelemetry/propagator-aws-xray` (production)

### `@oshun/health`

Provides standardized health check endpoints:

- `GET /health` - Basic liveness check
- `GET /ready` - Readiness check (includes dependency verification)

---

## Production Monitoring

### AWS CloudWatch

Production services log to CloudWatch Logs and emit CloudWatch Metrics via the
`@aws-sdk/client-cloudwatch` SDK.

Terraform modules in `infra/terraform/cloudwatch/` configure:

- Log groups per service
- Metric alarms
- Dashboard definitions

### Iris Domain: Production Dashboards

The Iris domain has the most comprehensive production monitoring setup, located
in `infra/monitoring/iris/`:

**Prometheus Configuration:**

- `prometheus/prometheus.yml` - Scrape targets
- `prometheus/recording-rules.yaml` - Pre-computed metrics
- `prometheus/servicemonitors.yaml` - Kubernetes ServiceMonitor CRDs

**Grafana Dashboards:**

| Dashboard                 | Purpose                          |
| ------------------------- | -------------------------------- |
| `iris-overview.json`      | System-wide health overview      |
| `iris-agent.json`         | AI agent performance metrics     |
| `iris-memory.json`        | Memory service utilization       |
| `iris-ai-providers.json`  | LLM provider latency and costs   |
| `iris-slo.json`           | Service level objective tracking |
| `iris-conversations.json` | Conversation analytics           |

**Alerting:**

- `alerting/alerts.yaml` - Alert rules for critical conditions
- Kustomize overlays for environment-specific thresholds

---

## Adding Monitoring to a Service

### 1. Add metrics

```typescript
import { createMetrics } from '@oshun/metrics';

const metrics = createMetrics({ serviceName: 'my-service' });

// Register custom metrics
const requestCounter = metrics.counter({
  name: 'my_service_requests_total',
  help: 'Total requests processed',
  labelNames: ['method', 'status'],
});
```

### 2. Add tracing

```typescript
import { initTracing } from '@oshun/tracing';

initTracing({
  serviceName: 'my-service',
  exporterUrl: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
});
```

### 3. Add health checks

```typescript
import { createHealthCheck } from '@oshun/health';

const health = createHealthCheck({
  checks: {
    database: () => db.ping(),
    redis: () => redis.ping(),
  },
});

app.get('/health', health.liveness);
app.get('/ready', health.readiness);
```

---

## Admin Tools

Additional admin tools are available via the `tools` Docker Compose profile:

```bash
docker compose -f docker/docker-compose.dev.yml --profile tools up -d
```

| Tool            | Port | Purpose          | Credentials           |
| --------------- | ---- | ---------------- | --------------------- |
| Redis Commander | 8081 | Redis GUI        | None                  |
| PgAdmin         | 5050 | PostgreSQL GUI   | admin@oshun.dev/admin |
| Kafka UI        | 8082 | Kafka management | None                  |

---

## Further Reading

- [Architecture](./architecture.md) - System architecture overview
- [CI/CD Pipelines](./ci-cd.md) - How monitoring integrates with CI
- [Getting Started](./getting-started.md) - Starting the observability stack