Monitoring and Observability

Overview#

Oshun uses a layered observability strategy: OpenTelemetry for instrumentation, Prometheus and Grafana for metrics and dashboards, Jaeger for distributed tracing, and CloudWatch for production AWS monitoring.

Development Environment#

Starting Observability Stack#

The observability services are available as a Docker Compose profile:

docker compose -f docker/docker-compose.dev.yml --profile observability up -d

This starts:

Prometheus#

URL: http://localhost:9090

Configuration file: docker/observability/prometheus.dev.yml

Prometheus scrapes metrics from all services exposing a /metrics endpoint. Services use @oshun/metrics (which wraps prom-client) to expose Prometheus metrics.

Grafana#

URL: http://localhost:3100

Default credentials: admin / admin

Grafana connects to Prometheus as a data source and provides pre-built dashboards. Dashboard provisioning configs are in docker/observability/grafana/provisioning/.

Jaeger#

URL: http://localhost:16686

Jaeger collects distributed traces via OpenTelemetry. Services use @oshun/tracing to instrument HTTP requests, database queries, and inter-service calls.

Ports:

• 16686 - UI
• 14268 - Collector (HTTP)
• 6831/udp - Agent (Thrift compact)
• 9411 - Zipkin-compatible endpoint

Instrumentation Libraries#

@oshun/metrics#

Wraps prom-client to provide:

• HTTP request duration histograms
• Request count counters
• Active connection gauges
• Custom domain-specific metrics
• /metrics endpoint middleware

@oshun/tracing#

Wraps OpenTelemetry SDK to provide:

• Automatic HTTP request tracing
• Database query instrumentation
• Inter-service propagation (W3C Trace Context)
• Custom span creation

OpenTelemetry packages used:

• @opentelemetry/api
• @opentelemetry/sdk-trace-node
• @opentelemetry/sdk-trace-base
• @opentelemetry/exporter-trace-otlp-http
• @opentelemetry/resources
• @opentelemetry/semantic-conventions
• @opentelemetry/context-async-hooks
• @opentelemetry/propagator-aws-xray (production)

@oshun/health#

Provides standardized health check endpoints:

• GET /health - Basic liveness check
• GET /ready - Readiness check (includes dependency verification)

Production Monitoring#

AWS CloudWatch#

Production services log to CloudWatch Logs and emit CloudWatch Metrics via the @aws-sdk/client-cloudwatch SDK.

Terraform modules in infra/terraform/cloudwatch/ configure:

• Log groups per service
• Metric alarms
• Dashboard definitions

Iris Domain: Production Dashboards#

The Iris domain has the most comprehensive production monitoring setup, located in infra/monitoring/iris/:

Prometheus Configuration:

• prometheus/prometheus.yml - Scrape targets
• prometheus/recording-rules.yaml - Pre-computed metrics
• prometheus/servicemonitors.yaml - Kubernetes ServiceMonitor CRDs

Grafana Dashboards:

Alerting:

• alerting/alerts.yaml - Alert rules for critical conditions
• Kustomize overlays for environment-specific thresholds

Adding Monitoring to a Service#

1. Add metrics#

import { createMetrics } from '@oshun/metrics';

const metrics = createMetrics({ serviceName: 'my-service' });

// Register custom metrics
const requestCounter = metrics.counter({
  name: 'my_service_requests_total',
  help: 'Total requests processed',
  labelNames: ['method', 'status'],
});

2. Add tracing#

import { initTracing } from '@oshun/tracing';

initTracing({
  serviceName: 'my-service',
  exporterUrl: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
});

3. Add health checks#

import { createHealthCheck } from '@oshun/health';

const health = createHealthCheck({
  checks: {
    database: () => db.ping(),
    redis: () => redis.ping(),
  },
});

app.get('/health', health.liveness);
app.get('/ready', health.readiness);

Admin Tools#

Additional admin tools are available via the tools Docker Compose profile:

docker compose -f docker/docker-compose.dev.yml --profile tools up -d

Further Reading#

• Architecture - System architecture overview
• CI/CD Pipelines - How monitoring integrates with CI
• Getting Started - Starting the observability stack