This document provides comprehensive documentation for the Lilith Conversation Service Streaming API, including SSE (Server-Sent Events) and WebSocket streaming for real-time chat responses.
Overview#
The Streaming API enables real-time chat interactions with:
- SSE Streaming: Token-by-token HTTP streaming with backpressure handling
- WebSocket Streaming: Full-duplex communication with heartbeat
- Adaptive Throttling: Automatic flow control based on client capacity
- Nagle Optimization: Efficient network utilization with batched writes
Table of Contents#
SSE Streaming#
Overview#
Server-Sent Events (SSE) provides unidirectional streaming from server to client
over HTTP. Enable streaming by setting options.stream: true in chat requests.
Endpoint#
Endpoint: POST /v1/chat
Request Body:
| Field | Type | Required | Description |
|---|---|---|---|
persona |
string | No | AI persona to use |
input.text |
string | Yes | User message text |
options.stream |
boolean | Yes | Must be true for streaming |
options.message_type |
string | No | Message type (default: user) |
conversation_id |
string | No | Existing conversation ID |
Request:
curl -X POST "https://api.lilith.app/v1/chat" \
-H "Authorization: Bearer <access_token>" \
-H "Content-Type: application/json" \
-H "Accept: text/event-stream" \
-d '{
"persona": "meditation_guide",
"input": { "text": "Guide me through a breathing exercise" },
"options": { "stream": true },
"conversation_id": "conv_123"
}'
Response Headers#
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache, no-transform
Connection: keep-alive
x-correlation-id: corr-abc123
Event Stream Format#
event: token
data: {"text": "Let's "}
event: token
data: {"text": "begin "}
event: token
data: {"text": "with "}
event: token
data: {"text": "deep "}
event: token
data: {"text": "breathing."}
event: citations
data: [{"source_uri": "https://example.org/wisdom", "span": {"start": 0, "end": 36}, "license": "PD", "confidence": 0.85}]
event: complete
data: {"conversation_id": "conv_123", "metrics": {"tokens_in": 8, "tokens_out": 5, "latency_ms": 250, "citations_count": 1}}
event: done
data: {"message_id": "msg-xyz789", "conversation_id": "conv_123", "metrics": {"tokens_in": 8, "tokens_out": 5, "latency_ms": 250}, "retry_info": {"attempts": 1, "total_duration_ms": 260, "exhausted": false}}
JavaScript Client Example#
const response = await fetch('https://api.lilith.app/v1/chat', {
method: 'POST',
headers: {
Authorization: 'Bearer <access_token>',
'Content-Type': 'application/json',
Accept: 'text/event-stream',
},
body: JSON.stringify({
persona: 'meditation_guide',
input: { text: 'Guide me through breathing' },
options: { stream: true },
}),
});
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop() || '';
for (const line of lines) {
if (line.startsWith('event: ')) {
const eventType = line.slice(7);
// Handle event type
} else if (line.startsWith('data: ')) {
const data = JSON.parse(line.slice(6));
if (data.text) {
// Append token to output
process.stdout.write(data.text);
}
}
}
}
WebSocket Streaming#
Overview#
WebSocket provides full-duplex communication for real-time chat with bidirectional messaging, heartbeat, and conversation history retrieval.
Connection#
Endpoint: wss://api.lilith.app/v1/ws/chat
const ws = new WebSocket('wss://api.lilith.app/v1/ws/chat');
ws.onopen = () => {
console.log('Connected');
};
ws.onmessage = (event) => {
const message = JSON.parse(event.data);
handleMessage(message);
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
};
ws.onclose = () => {
console.log('Disconnected');
};
Connection Response#
Upon connection, the server sends a connection confirmation:
{
"type": "connection",
"data": {
"connection_id": "ws-abc123",
"status": "connected",
"server_info": {
"service": "conversation",
"version": "1.0"
}
}
}
Client-to-Server Messages#
Chat Message#
Send a chat message for processing:
{
"type": "chat",
"persona": "meditation_guide",
"input": { "text": "How do I meditate?" },
"conversation_id": "conv_123",
"options": {
"stream": true,
"message_type": "user"
}
}
Ping (Heartbeat)#
Send a ping to keep the connection alive:
{
"type": "ping"
}
History Request#
Request conversation history:
{
"type": "history",
"conversation_id": "conv_123"
}
Server-to-Client Messages#
Message Received#
Acknowledgment that a message was received:
{
"type": "message_received",
"data": {
"conversation_id": "conv_123",
"message_id": "msg-xyz789",
"persona": "meditation_guide",
"options": { "message_type": "user" },
"received_at": "2024-01-01T00:00:00.000Z"
}
}
Start Event#
Indicates response generation has started:
{
"type": "start",
"data": {
"conversation_id": "conv_123",
"message_id": "msg-xyz789",
"persona": "meditation_guide",
"options": { "message_type": "user" }
}
}
Token Event#
Individual token/word in the response:
{
"type": "token",
"data": {
"text": "meditation ",
"sequence": 5,
"message_id": "msg-xyz789"
}
}
Response Complete#
Full response completed:
{
"type": "response_complete",
"data": {
"conversation_id": "conv_123",
"message_id": "msg-xyz789",
"persona": "meditation_guide",
"full_text": "(WebSocket) reply to: How do I meditate?",
"citations": [],
"options": { "message_type": "user" },
"metrics": {
"tokens_in": 5,
"tokens_out": 8,
"latency_ms": 150
}
}
}
Pong (Heartbeat Response)#
Response to ping:
{
"type": "pong",
"data": {
"timestamp": "2024-01-01T00:00:00.000Z"
}
}
History Response#
Conversation history:
{
"type": "history",
"data": {
"conversation_id": "conv_123",
"messages": [
{
"id": "msg-001",
"role": "user",
"content": "Hello",
"created_at": "2024-01-01T00:00:00.000Z"
}
]
}
}
Error Event#
Error notification:
{
"type": "error",
"data": {
"code": "empty_message",
"message": "Empty message text is not allowed"
}
}
WebSocket Status Endpoint#
Check active WebSocket connections:
Endpoint: GET /v1/ws/status
Response:
{
"connected_clients": 5,
"connections": [
{
"connection_id": "ws-abc123",
"metadata": {
"connected_at": "2024-01-01T00:00:00.000Z",
"client_ip": "192.168.1.1",
"user_agent": "Mozilla/5.0..."
}
}
]
}
Event Types#
SSE Event Types#
| Event | Description |
|---|---|
token |
Individual token/word in response |
citations |
Citation references for the response |
complete |
Response generation complete with metrics |
done |
Final event with full metadata |
error |
Error during streaming |
WebSocket Message Types#
| Type | Direction | Description |
|---|---|---|
connection |
S→C | Connection established |
chat |
C→S | Send chat message |
message_received |
S→C | Message acknowledged |
start |
S→C | Response generation starting |
token |
S→C | Individual token |
response_complete |
S→C | Full response ready |
complete |
S→C | Final completion event |
ping |
C→S | Heartbeat request |
pong |
S→C | Heartbeat response |
history |
Both | Request/receive history |
error |
S→C | Error notification |
Backpressure Handling#
SSE Backpressure#
The server implements adaptive backpressure handling:
- Buffer Monitoring: Tracks buffer size (max 64KB)
- Adaptive Delay: Adjusts throttle based on write success
- Drain Events: Pauses on buffer pressure, resumes on drain
- Nagle Batching: Combines small writes for efficiency
Configuration:
maxBufferSize: 64KB
baseThrottleDelay: 50ms
adaptiveMultiplier: 1.0 - 5.0x
nagleMaxChunkSize: 8KB
nagleFlushInterval: 100ms
WebSocket Backpressure#
WebSocket implements similar adaptive handling:
maxBufferSize: 32KB
baseThrottleDelay: 100ms
adaptiveMultiplier: 1.0 - 4.0x
When bufferedAmount exceeds threshold, the server:
- Increases delay multiplier (up to 4x)
- Waits for buffer to drain below 50%
- Gradually reduces delay as pressure eases
Error Handling#
SSE Errors#
Errors are sent as events before stream ends:
event: error
data: {"code": "chat.streaming_error", "message": "Error during streaming", "details": "Connection timeout"}
WebSocket Errors#
Errors are sent as messages:
{
"type": "error",
"data": {
"code": "invalid_message",
"message": "Failed to parse message payload",
"details": "Unexpected token in JSON"
}
}
Error Codes#
| Code | Description |
|---|---|
empty_message |
Message text is empty |
invalid_message |
Invalid message format |
unknown_message_type |
Unrecognized message type |
chat.streaming_error |
Error during stream processing |
rate_limit_exceeded |
Too many requests |
circuit_breaker_open |
Service temporarily unavailable |
SDK Integration#
TypeScript/JavaScript#
import { LilithClient } from '@lilith/sdk';
const client = new LilithClient({
apiKey: process.env.LILITH_API_KEY,
});
// SSE Streaming
const stream = await client.conversation.chatStream({
persona: 'meditation_guide',
text: 'Guide me through breathing',
conversationId: 'conv_123',
});
for await (const event of stream) {
if (event.type === 'token') {
process.stdout.write(event.data.text);
} else if (event.type === 'complete') {
console.log('\n\nMetrics:', event.data.metrics);
}
}
// WebSocket
const ws = client.conversation.connectWebSocket();
ws.on('connected', (data) => {
console.log('Connected:', data.connection_id);
});
ws.on('token', (data) => {
process.stdout.write(data.text);
});
ws.on('response_complete', (data) => {
console.log('\n\nComplete:', data.full_text);
});
// Send message
ws.send({
type: 'chat',
persona: 'meditation_guide',
input: { text: 'How do I meditate?' },
conversation_id: 'conv_123',
});
// Heartbeat
setInterval(() => {
ws.send({ type: 'ping' });
}, 30000);
Related Documentation#
- Thread API - Thread and message management
- Chat API - Non-streaming chat
- Context API - Context window management
- Authentication - API authentication
Questions?#
- Technical Support: api-support@lilith.app
- Documentation: GitHub Issues