# Distributed Tracing Standards

Comprehensive guide for implementing distributed tracing across the Lilith
platform.

## Table of Contents

- [Overview](#overview)
- [Tracing Architecture](#tracing-architecture)
- [Implementation Guide](#implementation-guide)
- [Instrumentation Standards](#instrumentation-standards)
- [Span Naming Conventions](#span-naming-conventions)
- [Context Propagation](#context-propagation)
- [Sampling Strategies](#sampling-strategies)
- [Correlation with Logs and Metrics](#correlation-with-logs-and-metrics)
- [Troubleshooting](#troubleshooting)

## Overview

Distributed tracing provides visibility into request flows across microservices,
enabling performance analysis and debugging.

### Tracing Stack

| Component     | Technology              | Purpose                     |
| ------------- | ----------------------- | --------------------------- |
| SDK           | @lilith/tracing         | Application instrumentation |
| Collector     | OpenTelemetry Collector | Trace aggregation           |
| Backend       | Tempo                   | Trace storage               |
| Visualization | Grafana                 | Trace exploration           |

### Trace Flow

```
┌─────────────────────────────────────────────────────────────────────────┐
│                          Request Flow                                    │
│                                                                          │
│  Client ──▶ BFF ──▶ Auth ──▶ Content ──▶ AI                            │
│    │         │        │         │        │                              │
│    ▼         ▼        ▼         ▼        ▼                              │
│  Span 1   Span 2   Span 3   Span 4   Span 5                            │
│  (root)   (child)  (child)  (child)  (child)                           │
│    │         │        │         │        │                              │
│    └─────────┴────────┴─────────┴────────┘                              │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │  OpenTelemetry      │                                    │
│              │  Collector          │                                    │
│              └─────────────────────┘                                    │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │       Tempo         │                                    │
│              │  (Trace Storage)    │                                    │
│              └─────────────────────┘                                    │
│                        │                                                 │
│                        ▼                                                 │
│              ┌─────────────────────┐                                    │
│              │      Grafana        │                                    │
│              │  (Visualization)    │                                    │
│              └─────────────────────┘                                    │
└─────────────────────────────────────────────────────────────────────────┘
```

## Tracing Architecture

### W3C Trace Context

All services use W3C Trace Context for distributed tracing:

```
traceparent: 00-{trace-id}-{span-id}-{trace-flags}

Example:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
                 └────────trace-id────────────┘ └───span-id──────┘ └flags
```

### Core Components

```typescript
// packages/tracing/src/index.ts

export interface SpanContext {
  traceId: string; // 32 hex chars
  spanId: string; // 16 hex chars
  traceFlags: number; // 1 = sampled
  traceState?: string;
}

export interface Span {
  name: string;
  context: SpanContext;
  parentSpanId?: string;
  startTime: number;
  endTime?: number;
  attributes: SpanAttributes;
  events: SpanEvent[];
  status: SpanStatus;
}
```

## Implementation Guide

### Basic Setup

```typescript
import { createTracer } from '@lilith/tracing';

// Create a tracer for your service
const tracer = createTracer({
  serviceName: 'auth-service',
  serviceVersion: '1.0.0',
  enabled: true,
  sampleRate: 1.0,
});

// Use in request handlers
app.get('/api/users/:id', async (request, reply) => {
  return tracer.startSpan('getUser', async (span) => {
    span.setAttribute('user.id', request.params.id);

    const user = await userService.findById(request.params.id);

    span.setAttribute('user.found', !!user);
    return user;
  });
});
```

### Fastify Integration

```typescript
import Fastify from 'fastify';
import {
  createTracer,
  extractTraceContext,
  injectTraceContext,
} from '@lilith/tracing';

const app = Fastify();
const tracer = createTracer('my-service');

// Add tracing hook
app.addHook('onRequest', async (request, reply) => {
  const parentContext = extractTraceContext(request.headers);

  request.traceContext = parentContext;
  request.span = tracer.startSpan(
    `${request.method} ${request.url}`,
    parentContext
  );

  request.span.setAttribute('http.method', request.method);
  request.span.setAttribute('http.url', request.url);
  request.span.setAttribute(
    'http.target',
    request.routeOptions?.url || request.url
  );
});

app.addHook('onResponse', async (request, reply) => {
  request.span.setAttribute('http.status_code', reply.statusCode);
  request.span.end();
});

app.addHook('onError', async (request, reply, error) => {
  request.span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });
  request.span.addEvent('exception', {
    'exception.type': error.name,
    'exception.message': error.message,
    'exception.stacktrace': error.stack,
  });
});
```

### Service-to-Service Calls

```typescript
import { injectTraceContext } from '@lilith/tracing';

async function callDownstreamService(tracer, parentSpan) {
  return tracer.startSpan('call-content-service', async (span) => {
    const headers = {};
    injectTraceContext(span.context, headers);

    const response = await fetch('http://content-service/api/data', {
      headers: {
        ...headers,
        'Content-Type': 'application/json',
      },
    });

    span.setAttribute('http.status_code', response.status);
    return response.json();
  });
}
```

## Instrumentation Standards

### Required Attributes

#### HTTP Spans

| Attribute          | Type   | Description                   |
| ------------------ | ------ | ----------------------------- |
| `http.method`      | string | HTTP method (GET, POST, etc.) |
| `http.url`         | string | Full URL                      |
| `http.target`      | string | Path and query string         |
| `http.status_code` | number | Response status code          |
| `http.host`        | string | Host header                   |
| `http.scheme`      | string | http or https                 |

```typescript
span.setAttributes({
  'http.method': 'GET',
  'http.url': 'https://api.lilith.app/v1/users/123',
  'http.target': '/v1/users/123',
  'http.status_code': 200,
  'http.host': 'api.lilith.app',
  'http.scheme': 'https',
});
```

#### Database Spans

| Attribute      | Type   | Description                       |
| -------------- | ------ | --------------------------------- |
| `db.system`    | string | Database type (postgresql, redis) |
| `db.name`      | string | Database name                     |
| `db.statement` | string | Query (sanitized)                 |
| `db.operation` | string | Operation type (query, insert)    |

```typescript
span.setAttributes({
  'db.system': 'postgresql',
  'db.name': 'lilith_auth',
  'db.statement': 'SELECT * FROM users WHERE id = $1',
  'db.operation': 'query',
});
```

#### gRPC Spans

| Attribute              | Type   | Description      |
| ---------------------- | ------ | ---------------- |
| `rpc.system`           | string | 'grpc'           |
| `rpc.service`          | string | Service name     |
| `rpc.method`           | string | Method name      |
| `rpc.grpc.status_code` | number | gRPC status code |

#### Messaging Spans

| Attribute               | Type   | Description            |
| ----------------------- | ------ | ---------------------- |
| `messaging.system`      | string | kafka, rabbitmq, etc.  |
| `messaging.destination` | string | Queue/topic name       |
| `messaging.operation`   | string | send, receive, process |

### Custom Business Attributes

```typescript
// User context
span.setAttribute('user.id', userId);
span.setAttribute('user.role', userRole);

// Content context
span.setAttribute('content.id', contentId);
span.setAttribute('content.type', 'meditation');

// AI context
span.setAttribute('ai.model', 'gpt-4');
span.setAttribute('ai.tokens_used', tokenCount);
```

## Span Naming Conventions

### HTTP Endpoints

```
{HTTP_METHOD} {PATH_TEMPLATE}

Examples:
GET /v1/users/{userId}
POST /v1/content/courses
DELETE /v1/sessions/{sessionId}
```

### Database Operations

```
{OPERATION} {TABLE/COLLECTION}

Examples:
SELECT users
INSERT content_items
UPDATE user_preferences
```

### Service Calls

```
{SERVICE_NAME}.{METHOD_NAME}

Examples:
auth.validateToken
content.getCourse
ai.generateResponse
```

### Background Jobs

```
job.{JOB_TYPE}

Examples:
job.processMediaUpload
job.sendNotification
job.syncUserData
```

## Context Propagation

### HTTP Headers

```typescript
// Inject context into outgoing request
const headers = {};
injectTraceContext(currentSpan.context, headers);
// Result: { 'traceparent': '00-trace-span-01' }

// Extract context from incoming request
const context = extractTraceContext(request.headers);
tracer.startSpanWithContext('operation', context, async (span) => {
  // Child span linked to parent trace
});
```

### Kafka Messages

```typescript
// Producer
const message = {
  key: userId,
  value: JSON.stringify(event),
  headers: {
    traceparent: `00-${span.context.traceId}-${span.context.spanId}-01`,
  },
};

// Consumer
const traceparent = message.headers.traceparent?.toString();
const context = parseTraceparent(traceparent);
tracer.startSpanWithContext('process-message', context, async (span) => {
  // Process message
});
```

### gRPC Metadata

```typescript
// Client
const metadata = new grpc.Metadata();
metadata.set('traceparent', formatTraceparent(span.context));

// Server
const traceparent = call.metadata.get('traceparent')[0];
```

## Sampling Strategies

### Head-Based Sampling

Sample decision made at trace start:

```typescript
const tracer = createTracer({
  serviceName: 'my-service',
  sampleRate: 0.1, // Sample 10% of traces
});
```

### Adaptive Sampling

```typescript
const tracer = createTracer({
  serviceName: 'my-service',
  sampleRate: calculateSampleRate(),
});

function calculateSampleRate(): number {
  const currentRPS = getRequestsPerSecond();

  if (currentRPS > 10000) return 0.01; // 1% at high load
  if (currentRPS > 1000) return 0.1; // 10% at medium load
  return 1.0; // 100% at low load
}
```

### Error-Based Sampling

Always sample errors:

```typescript
app.addHook('onError', async (request, reply, error) => {
  // Force sample on error
  request.span.context.traceFlags = 1;

  request.span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });
});
```

### High-Value Request Sampling

```typescript
// Always sample specific operations
if (request.url.includes('/payment') || request.url.includes('/checkout')) {
  span.context.traceFlags = 1; // Force sample
}
```

## Correlation with Logs and Metrics

### Log Correlation

```typescript
import { logger } from '@lilith/logger';

app.addHook('onRequest', (request, reply) => {
  // Add trace context to logger
  request.log = logger.child({
    traceId: request.span?.context.traceId,
    spanId: request.span?.context.spanId,
  });
});

// Usage in handlers
request.log.info('Processing request');
// Output: {"message":"Processing request","traceId":"abc123","spanId":"def456"}
```

### Metric Correlation

```typescript
import { histogram } from '@lilith/metrics';

const requestDuration = histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'status', 'trace_sampled'],
});

app.addHook('onResponse', (request, reply) => {
  requestDuration.observe(
    {
      method: request.method,
      status: reply.statusCode,
      trace_sampled: request.span?.context.traceFlags === 1 ? 'true' : 'false',
    },
    reply.getResponseTime() / 1000
  );
});
```

### Exemplars (Linking Metrics to Traces)

```typescript
// Add trace ID as exemplar to histogram
requestDuration.observe({ method: 'GET', status: '200' }, duration, {
  traceId: span.context.traceId,
});
```

## Troubleshooting

### Missing Traces

1. **Check sampling rate**

   ```typescript
   // Temporarily set to 100%
   sampleRate: 1.0;
   ```

2. **Verify context propagation**

   ```bash
   # Check incoming headers
   curl -v -H "traceparent: 00-abc123-def456-01" http://localhost:3000/api/test
   ```

3. **Check collector connectivity**
   ```bash
   kubectl logs -n monitoring deploy/otel-collector
   ```

### Broken Traces

1. **Context not propagated**

   ```typescript
   // Ensure headers are forwarded
   const headers = {};
   injectTraceContext(span.context, headers);
   ```

2. **Async context lost**
   ```typescript
   // Use context management
   const activeSpan = tracer.getActiveSpan();
   ```

### Performance Issues

1. **Too many attributes**

   ```typescript
   // Limit attribute count
   const MAX_ATTRIBUTES = 32;
   ```

2. **Large payloads**
   ```typescript
   // Don't trace large request/response bodies
   span.setAttribute('http.request_content_length', body.length);
   // NOT: span.setAttribute('http.request_body', body);
   ```

### Debug Commands

```bash
# Check trace in Tempo
curl -G http://tempo:3200/api/traces/{traceId}

# Query Tempo via Grafana
# TraceQL: {service.name="auth-service" && status=error}

# View recent traces
kubectl port-forward -n monitoring svc/tempo 3200:3200
curl http://localhost:3200/api/traces?service=auth-service&limit=10
```

## Related Documentation

- [Logging Standards](../LOGGING_STANDARDS.md) - Structured logging
- [Metrics Standards](../runbooks/procedures/monitoring.md) - Prometheus metrics
- [Observability Infrastructure](../infrastructure/eks.md#observability) - Tempo
  deployment
