Eve library

Streaming API

Guide

Revision 1

On this page

This document provides comprehensive documentation for the Lilith Conversation Service Streaming API, including SSE (Server-Sent Events) and WebSocket streaming for real-time chat responses.

Overview#

The Streaming API enables real-time chat interactions with:

  • SSE Streaming: Token-by-token HTTP streaming with backpressure handling
  • WebSocket Streaming: Full-duplex communication with heartbeat
  • Adaptive Throttling: Automatic flow control based on client capacity
  • Nagle Optimization: Efficient network utilization with batched writes

Table of Contents#

  1. SSE Streaming
  2. WebSocket Streaming
  3. Event Types
  4. Backpressure Handling
  5. Error Handling

SSE Streaming#

Overview#

Server-Sent Events (SSE) provides unidirectional streaming from server to client over HTTP. Enable streaming by setting options.stream: true in chat requests.

Endpoint#

Endpoint: POST /v1/chat

Request Body:

Field Type Required Description
persona string No AI persona to use
input.text string Yes User message text
options.stream boolean Yes Must be true for streaming
options.message_type string No Message type (default: user)
conversation_id string No Existing conversation ID

Request:

bash
curl -X POST "https://api.lilith.app/v1/chat" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "persona": "meditation_guide",
    "input": { "text": "Guide me through a breathing exercise" },
    "options": { "stream": true },
    "conversation_id": "conv_123"
  }'

Response Headers#

text
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
x-correlation-id: corr-abc123

Event Stream Format#

text
event: token
data: {"text": "Let's "}

event: token
data: {"text": "begin "}

event: token
data: {"text": "with "}

event: token
data: {"text": "deep "}

event: token
data: {"text": "breathing."}

event: citations
data: [{"source_uri": "https://example.org/wisdom", "span": {"start": 0, "end": 36}, "license": "PD", "confidence": 0.85}]

event: complete
data: {"conversation_id": "conv_123", "metrics": {"tokens_in": 8, "tokens_out": 5, "latency_ms": 250, "citations_count": 1}}

event: done
data: {"message_id": "msg-xyz789", "conversation_id": "conv_123", "metrics": {"tokens_in": 8, "tokens_out": 5, "latency_ms": 250}, "retry_info": {"attempts": 1, "total_duration_ms": 260, "exhausted": false}}

JavaScript Client Example#

javascript
const response = await fetch('https://api.lilith.app/v1/chat', {
  method: 'POST',
  headers: {
    Authorization: 'Bearer <access_token>',
    'Content-Type': 'application/json',
    Accept: 'text/event-stream',
  },
  body: JSON.stringify({
    persona: 'meditation_guide',
    input: { text: 'Guide me through breathing' },
    options: { stream: true },
  }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop() || '';

  for (const line of lines) {
    if (line.startsWith('event: ')) {
      const eventType = line.slice(7);
      // Handle event type
    } else if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      if (data.text) {
        // Append token to output
        process.stdout.write(data.text);
      }
    }
  }
}

WebSocket Streaming#

Overview#

WebSocket provides full-duplex communication for real-time chat with bidirectional messaging, heartbeat, and conversation history retrieval.

Connection#

Endpoint: wss://api.lilith.app/v1/ws/chat

javascript
const ws = new WebSocket('wss://api.lilith.app/v1/ws/chat');

ws.onopen = () => {
  console.log('Connected');
};

ws.onmessage = (event) => {
  const message = JSON.parse(event.data);
  handleMessage(message);
};

ws.onerror = (error) => {
  console.error('WebSocket error:', error);
};

ws.onclose = () => {
  console.log('Disconnected');
};

Connection Response#

Upon connection, the server sends a connection confirmation:

json
{
  "type": "connection",
  "data": {
    "connection_id": "ws-abc123",
    "status": "connected",
    "server_info": {
      "service": "conversation",
      "version": "1.0"
    }
  }
}

Client-to-Server Messages#

Chat Message#

Send a chat message for processing:

json
{
  "type": "chat",
  "persona": "meditation_guide",
  "input": { "text": "How do I meditate?" },
  "conversation_id": "conv_123",
  "options": {
    "stream": true,
    "message_type": "user"
  }
}

Ping (Heartbeat)#

Send a ping to keep the connection alive:

json
{
  "type": "ping"
}

History Request#

Request conversation history:

json
{
  "type": "history",
  "conversation_id": "conv_123"
}

Server-to-Client Messages#

Message Received#

Acknowledgment that a message was received:

json
{
  "type": "message_received",
  "data": {
    "conversation_id": "conv_123",
    "message_id": "msg-xyz789",
    "persona": "meditation_guide",
    "options": { "message_type": "user" },
    "received_at": "2024-01-01T00:00:00.000Z"
  }
}

Start Event#

Indicates response generation has started:

json
{
  "type": "start",
  "data": {
    "conversation_id": "conv_123",
    "message_id": "msg-xyz789",
    "persona": "meditation_guide",
    "options": { "message_type": "user" }
  }
}

Token Event#

Individual token/word in the response:

json
{
  "type": "token",
  "data": {
    "text": "meditation ",
    "sequence": 5,
    "message_id": "msg-xyz789"
  }
}

Response Complete#

Full response completed:

json
{
  "type": "response_complete",
  "data": {
    "conversation_id": "conv_123",
    "message_id": "msg-xyz789",
    "persona": "meditation_guide",
    "full_text": "(WebSocket) reply to: How do I meditate?",
    "citations": [],
    "options": { "message_type": "user" },
    "metrics": {
      "tokens_in": 5,
      "tokens_out": 8,
      "latency_ms": 150
    }
  }
}

Pong (Heartbeat Response)#

Response to ping:

json
{
  "type": "pong",
  "data": {
    "timestamp": "2024-01-01T00:00:00.000Z"
  }
}

History Response#

Conversation history:

json
{
  "type": "history",
  "data": {
    "conversation_id": "conv_123",
    "messages": [
      {
        "id": "msg-001",
        "role": "user",
        "content": "Hello",
        "created_at": "2024-01-01T00:00:00.000Z"
      }
    ]
  }
}

Error Event#

Error notification:

json
{
  "type": "error",
  "data": {
    "code": "empty_message",
    "message": "Empty message text is not allowed"
  }
}

WebSocket Status Endpoint#

Check active WebSocket connections:

Endpoint: GET /v1/ws/status

Response:

json
{
  "connected_clients": 5,
  "connections": [
    {
      "connection_id": "ws-abc123",
      "metadata": {
        "connected_at": "2024-01-01T00:00:00.000Z",
        "client_ip": "192.168.1.1",
        "user_agent": "Mozilla/5.0..."
      }
    }
  ]
}

Event Types#

SSE Event Types#

Event Description
token Individual token/word in response
citations Citation references for the response
complete Response generation complete with metrics
done Final event with full metadata
error Error during streaming

WebSocket Message Types#

Type Direction Description
connection S→C Connection established
chat C→S Send chat message
message_received S→C Message acknowledged
start S→C Response generation starting
token S→C Individual token
response_complete S→C Full response ready
complete S→C Final completion event
ping C→S Heartbeat request
pong S→C Heartbeat response
history Both Request/receive history
error S→C Error notification

Backpressure Handling#

SSE Backpressure#

The server implements adaptive backpressure handling:

  1. Buffer Monitoring: Tracks buffer size (max 64KB)
  2. Adaptive Delay: Adjusts throttle based on write success
  3. Drain Events: Pauses on buffer pressure, resumes on drain
  4. Nagle Batching: Combines small writes for efficiency

Configuration:

text
maxBufferSize: 64KB
baseThrottleDelay: 50ms
adaptiveMultiplier: 1.0 - 5.0x
nagleMaxChunkSize: 8KB
nagleFlushInterval: 100ms

WebSocket Backpressure#

WebSocket implements similar adaptive handling:

text
maxBufferSize: 32KB
baseThrottleDelay: 100ms
adaptiveMultiplier: 1.0 - 4.0x

When bufferedAmount exceeds threshold, the server:

  1. Increases delay multiplier (up to 4x)
  2. Waits for buffer to drain below 50%
  3. Gradually reduces delay as pressure eases

Error Handling#

SSE Errors#

Errors are sent as events before stream ends:

text
event: error
data: {"code": "chat.streaming_error", "message": "Error during streaming", "details": "Connection timeout"}

WebSocket Errors#

Errors are sent as messages:

json
{
  "type": "error",
  "data": {
    "code": "invalid_message",
    "message": "Failed to parse message payload",
    "details": "Unexpected token in JSON"
  }
}

Error Codes#

Code Description
empty_message Message text is empty
invalid_message Invalid message format
unknown_message_type Unrecognized message type
chat.streaming_error Error during stream processing
rate_limit_exceeded Too many requests
circuit_breaker_open Service temporarily unavailable

SDK Integration#

TypeScript/JavaScript#

typescript
import { LilithClient } from '@lilith/sdk';

const client = new LilithClient({
  apiKey: process.env.LILITH_API_KEY,
});

// SSE Streaming
const stream = await client.conversation.chatStream({
  persona: 'meditation_guide',
  text: 'Guide me through breathing',
  conversationId: 'conv_123',
});

for await (const event of stream) {
  if (event.type === 'token') {
    process.stdout.write(event.data.text);
  } else if (event.type === 'complete') {
    console.log('\n\nMetrics:', event.data.metrics);
  }
}

// WebSocket
const ws = client.conversation.connectWebSocket();

ws.on('connected', (data) => {
  console.log('Connected:', data.connection_id);
});

ws.on('token', (data) => {
  process.stdout.write(data.text);
});

ws.on('response_complete', (data) => {
  console.log('\n\nComplete:', data.full_text);
});

// Send message
ws.send({
  type: 'chat',
  persona: 'meditation_guide',
  input: { text: 'How do I meditate?' },
  conversation_id: 'conv_123',
});

// Heartbeat
setInterval(() => {
  ws.send({ type: 'ping' });
}, 30000);


Questions?#

  • Technical Support: api-support@lilith.app
  • Documentation: GitHub Issues