This document provides a comprehensive inventory of all external API
integrations used by the Serwaa platform (source codebase for the Psyche domain
migration).
Table of Contents#
Overview
AI/LLM APIs
Voice/Audio APIs
Avatar/3D Generation APIs
Video Conferencing SDKs
Embeddings & Vector Database
Browser Automation
Mock Services
Integration Patterns
Environment Variables Reference
Migration Considerations
Overview#
The Serwaa platform integrates with 50+ external APIs across 8 major categories:
Category
Primary Provider
Fallback Providers
Integration Type
LLM/AI
Anthropic Claude
OpenAI, Google Gemini, Local LLM
REST, Streaming
Text-to-Speech
ElevenLabs
Cartesia, OpenAI, Deepgram Aura
WebSocket, REST
Speech-to-Text
Deepgram
Whisper, AssemblyAI
WebSocket, REST
Avatar Rendering
3D Gaussian Splatting
Tavus (optional)
Local, REST
Video Conferencing
Multi-platform
-
SDK, Browser, WebRTC
Embeddings
Voyage AI
Cohere, OpenAI
REST
Vector Database
Qdrant
-
REST
Browser Automation
Playwright
-
Library
Integration Architecture#
text Copy
┌─────────────────────────────────────────────────────────────────┐
│ Provider Chain Pattern │
├─────────────────────────────────────────────────────────────────┤
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Primary │→ │ Fallback │→ │ Fallback │ │
│ │ Provider │ │ Provider │ │ Provider │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ ↓ ↓ ↓ │
│ ┌────────────────────────────────────────────────────┐ │
│ │ Unified Response Interface │ │
│ └────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
AI/LLM APIs#
Anthropic Claude API#
Primary LLM provider for conversation and reasoning.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
serwaa/services/knowledge-base/src/knowledge_base/rag/generator.py
serwaa/services/computer-use/src/computer_use/agent/loop.py
Implementation Details#
Property
Value
Base URL
https://api.anthropic.com (configurable)
API Version
2023-06-01
Endpoint
POST /v1/messages
Auth Type
API Key (x-api-key header)
SDK
anthropic Python package
Supported Models#
Model ID
Context Window
Use Case
claude-3-5-sonnet-20241022
200K tokens
Default, balanced
claude-3-opus-20240229
200K tokens
Complex reasoning
claude-3-sonnet-20240229
200K tokens
Balanced
claude-3-haiku-20240307
200K tokens
Fast, cost-effective
Features#
Streaming Responses : Server-sent events via messages.stream()
Tool Calling : Function definitions via tools parameter
System Prompts : Persona and behavior configuration
Token Counting : Native token counting for context management
Vision : Image understanding in messages (for Computer Use)
Request Schema#
python Copy
{
"model": "claude-3-5-sonnet-20241022",
"max_tokens": 4096,
"system": "You are a helpful assistant...",
"messages": [
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hi there!"},
{"role": "user", "content": "What is the weather?"}
],
"tools": [
{
"name": "get_weather",
"description": "Get current weather for a location",
"input_schema": {
"type": "object",
"properties": {
"location": {"type": "string"}
},
"required": ["location"]
}
}
],
"stream": True
}
Response Schema (Streaming)#
python Copy
# Message start event
{"type": "message_start", "message": {"id": "msg_...", "role": "assistant"}}
# Content delta events
{"type": "content_block_delta", "delta": {"type": "text_delta", "text": "Hello"}}
# Tool use events
{"type": "content_block_start", "content_block": {"type": "tool_use", "name": "get_weather"}}
# Message end event
{"type": "message_stop"}
Error Handling#
Error Type
HTTP Code
Handling
RateLimitError
429
Exponential backoff, switch provider
InvalidRequestError
400
Context truncation if too long
APIConnectionError
-
Retry with backoff
AuthenticationError
401
Fail immediately
Configuration#
bash Copy
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
ANTHROPIC_MAX_TOKENS=4096
ANTHROPIC_BASE_URL=https://api.anthropic.com
OpenAI GPT API#
Fallback LLM provider and TTS/embeddings source.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
serwaa/services/voice-engine/src/voice_engine/tts/openai_tts.py
Implementation Details#
Property
Value
Base URL
https://api.openai.com/v1 (configurable)
Endpoint (Chat)
POST /chat/completions
Endpoint (TTS)
POST /audio/speech
Endpoint (Embeddings)
POST /embeddings
Auth Type
Bearer Token
SDK
openai Python package
Supported Models#
Model ID
Context Window
Use Case
gpt-4-turbo-preview
128K tokens
Default fallback
gpt-4
8K tokens
Legacy
gpt-3.5-turbo
16K tokens
Cost-effective
Text-to-Speech#
Property
Value
Model
tts-1-hd
Voices
alloy, echo, fable, onyx, nova, shimmer
Output Format
MP3, AAC, FLAC, WAV
Sample Rate
16000-24000 Hz
Features#
Streaming : SSE via stream=True
Tool Calling : Function definitions
JSON Mode : Structured output via response_format
Token Counting : Via tiktoken library
Configuration#
bash Copy
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4-turbo-preview
OPENAI_BASE_URL=https://api.openai.com/v1
Google Gemini API#
Alternative LLM provider with multimodal capabilities.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
Implementation Details#
Property
Value
Base URL
https://generativelanguage.googleapis.com/v1beta
Endpoint (Generate)
POST /models/{model}:generateContent
Endpoint (Stream)
POST /models/{model}:streamGenerateContent
Endpoint (Count)
POST /models/{model}:countTokens
Auth Type
API Key (query parameter)
Supported Models#
Model ID
Context Window
Use Case
gemini-1.5-pro
1M tokens
Long context
gemini-1.5-flash
1M tokens
Fast inference
gemini-pro
32K tokens
Standard
Features#
System Instructions : Via systemInstruction field
Function Calling : Via tools.functionDeclarations
Streaming : Server-sent events
Token Counting : Native endpoint
Configuration#
bash Copy
GOOGLE_API_KEY=AIza...
GEMINI_MODEL=gemini-1.5-pro
Local LLM (Ollama/vLLM/llama.cpp)#
Self-hosted LLM for offline or cost-sensitive deployments.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
Implementation Details#
Property
Value
Base URL
Configurable (default: http://localhost:11434/v1)
Endpoint
POST /chat/completions (OpenAI-compatible)
Auth Type
Optional Bearer Token
Servers
Ollama, vLLM, llama.cpp
Supported Models#
Model
Parameters
Use Case
llama3.1
8B-70B
General purpose
mistral
7B
Fast inference
mixtral
8x7B
MoE model
Configuration#
bash Copy
LOCAL_LLM_BASE_URL=http://localhost:11434/v1
LOCAL_LLM_MODEL=llama3.1
LOCAL_LLM_API_KEY= # Optional
LLM Provider Chain#
Orchestrates multiple LLM providers with failover and routing.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
Class: LLMProviderChain#
python Copy
class LLMProviderChain:
"""Manages multiple LLM providers with failover and routing."""
def __init__(
self,
primary_provider: str = "anthropic",
fallback_providers: List[str] = ["openai", "google"],
max_retries: int = 2,
cost_routing_enabled: bool = False,
ensemble_enabled: bool = False,
cache_enabled: bool = False
):
...
async def generate(
self,
messages: List[Message],
tools: Optional[List[Tool]] = None,
**kwargs
) -> Response:
"""Generate with automatic failover."""
...
async def generate_stream(
self,
messages: List[Message],
**kwargs
) -> AsyncIterator[StreamEvent]:
"""Stream generation with automatic failover."""
...
Features#
Feature
Description
Environment Variable
Failover
Switch providers on failure
LLM_FALLBACK_PROVIDERS
Cost Routing
Route based on cost/token
LLM_COST_ROUTING_ENABLED
Ensemble
Run multiple in parallel
LLM_ENSEMBLE_ENABLED
Caching
TTL-based response cache
LLM_RESPONSE_CACHE_ENABLED
Retry
Exponential backoff
LLM_MAX_RETRIES
Configuration#
bash Copy
LLM_PRIMARY_PROVIDER=anthropic
LLM_FALLBACK_PROVIDERS=openai,google
LLM_MAX_RETRIES=2
LLM_TIMEOUT_SECONDS=60
LLM_COST_ROUTING_ENABLED=false
LLM_ENSEMBLE_ENABLED=false
LLM_RESPONSE_CACHE_ENABLED=false
Voice/Audio APIs#
ElevenLabs TTS#
Primary text-to-speech provider with ultra-low latency streaming.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/tts/elevenlabs.py
Implementation Details#
Property
Value
WebSocket URL
wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input
HTTP Base URL
https://api.elevenlabs.io/v1
Auth Type
API Key (xi-api-key header)
Latency
~75ms first byte
Models#
Model ID
Latency
Quality
Use Case
eleven_flash_v2_5
Ultra-low
Good
Real-time
eleven_turbo_v2_5
Low
Better
Balanced
eleven_multilingual_v2
Medium
Best
Multilingual
Voice Configuration#
python Copy
{
"voice_id": "EXAVITQu4vU6e1aXChow", # Bella (default)
"model_id": "eleven_flash_v2_5",
"voice_settings": {
"stability": 0.5,
"similarity_boost": 0.75,
"style": 0.0,
"use_speaker_boost": True
},
"output_format": "pcm_24000" # PCM 24kHz
}
Format
Sample Rate
Use Case
pcm_16000
16 kHz
Low bandwidth
pcm_22050
22.05 kHz
Standard
pcm_24000
24 kHz
High quality
pcm_44100
44.1 kHz
Maximum quality
mp3_44100_128
44.1 kHz
Compressed
WebSocket Protocol#
python Copy
# Connection
ws = await websocket.connect(
f"wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input"
f"?model_id={model_id}&output_format={output_format}"
)
# Send text chunks
await ws.send(json.dumps({
"text": "Hello ",
"voice_settings": {...},
"xi_api_key": api_key
}))
# Receive audio chunks
async for message in ws:
data = json.loads(message)
if "audio" in data:
audio_bytes = base64.b64decode(data["audio"])
yield audio_bytes
# End stream
await ws.send(json.dumps({"text": ""}))
Configuration#
bash Copy
ELEVENLABS_API_KEY=...
ELEVENLABS_MODEL=eleven_flash_v2_5
ELEVENLABS_VOICE_ID=EXAVITQu4vU6e1aXChow
Cartesia Sonic TTS#
Alternative low-latency TTS provider.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/tts/cartesia.py
Implementation Details#
Property
Value
Base URL
https://api.cartesia.ai
Endpoint (Batch)
POST /v1/tts
Endpoint (Stream)
POST /v1/tts/stream
Auth Type
Bearer Token
Request Schema#
python Copy
{
"model": "sonic",
"text": "Hello world",
"voice_id": "...",
"output_format": "pcm_16bit",
"sample_rate": 24000,
"language": "en"
}
Features#
Streaming : Chunked audio response
Voice Cloning : Custom voice support
Multi-language : English, Spanish, French, etc.
Configuration#
bash Copy
CARTESIA_API_KEY=...
CARTESIA_BASE_URL=https://api.cartesia.ai
Deepgram STT#
Primary speech-to-text provider with real-time WebSocket streaming.
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/stt/deepgram.py
Implementation Details#
Property
Value
WebSocket URL
wss://api.deepgram.com/v1/listen
Auth Type
Token (Authorization: Token {api_key})
Latency
~100-300ms
Query Parameters#
Parameter
Default
Description
model
nova-2
Transcription model
encoding
linear16
Audio encoding
sample_rate
16000
Sample rate in Hz
channels
1
Audio channels
punctuate
true
Add punctuation
smart_format
true
Smart formatting
interim_results
true
Send partial results
utterance_end_ms
1000
Silence threshold
diarize
false
Speaker identification
language
en
Language code
Models#
Model
Latency
Accuracy
Use Case
nova-2
Low
Best
Default
nova
Low
Good
Fallback
enhanced
Medium
Better
Non-realtime
base
Low
Basic
Cost-effective
WebSocket Protocol#
python Copy
# Connection
ws = await websocket.connect(
f"wss://api.deepgram.com/v1/listen?"
f"model=nova-2&encoding=linear16&sample_rate=16000&"
f"punctuate=true&interim_results=true"
)
ws.extra_headers = {"Authorization": f"Token {api_key}"}
# Send audio chunks
await ws.send(audio_bytes)
# Receive transcription
async for message in ws:
data = json.loads(message)
if data["type"] == "Results":
transcript = data["channel"]["alternatives"][0]["transcript"]
is_final = data["is_final"]
# Word-level details
for word in data["channel"]["alternatives"][0]["words"]:
text = word["word"]
start = word["start"]
end = word["end"]
confidence = word["confidence"]
speaker = word.get("speaker") # If diarization enabled
# Keep-alive (every 10 seconds)
await ws.send(json.dumps({"type": "KeepAlive"}))
# Close stream
await ws.send(json.dumps({"type": "CloseStream"}))
Response Schema#
python Copy
{
"type": "Results",
"channel_index": [0, 1],
"duration": 1.5,
"start": 0.0,
"is_final": True,
"speech_final": True,
"channel": {
"alternatives": [
{
"transcript": "Hello world",
"confidence": 0.98,
"words": [
{"word": "Hello", "start": 0.0, "end": 0.5, "confidence": 0.99},
{"word": "world", "start": 0.6, "end": 1.0, "confidence": 0.97}
]
}
]
}
}
Configuration#
bash Copy
DEEPGRAM_API_KEY=...
STT_MODEL=nova-2
STT_SAMPLE_RATE=16000
OpenAI Whisper STT#
Fallback speech-to-text (local or API-based).
Source Files#
text Copy
serwaa/services/voice-engine/src/voice_engine/stt/whisper.py
Implementation Details#
Property
Value
API Endpoint
POST /v1/audio/transcriptions
Local
openai-whisper Python package
Auth Type
Bearer Token (API)
Models (Local)#
Model
Parameters
VRAM
Speed
tiny
39M
~1GB
Fastest
base
74M
~1GB
Fast
small
244M
~2GB
Balanced
medium
769M
~5GB
Good
large
1550M
~10GB
Best
Configuration#
bash Copy
# API mode
OPENAI_API_KEY=...
WHISPER_MODEL=whisper-1
# Local mode
WHISPER_MODEL_SIZE=medium
WHISPER_DEVICE=cuda # or cpu
Voice Provider Chain#
Orchestrates TTS and STT providers with failover.
Configuration#
bash Copy
# TTS Configuration
TTS_PRIMARY_PROVIDER=elevenlabs
TTS_FALLBACK_PROVIDERS=cartesia,openai
TTS_SAMPLE_RATE=24000
# STT Configuration
STT_PRIMARY_PROVIDER=deepgram
STT_FALLBACK_PROVIDERS=whisper,assemblyai
STT_SAMPLE_RATE=16000
Additional TTS Providers#
Provider
Type
Environment Variables
Orpheus
Self-hosted
ORPHEUS_TTS_BASE_URL
F5-TTS
Voice cloning
F5_TTS_BASE_URL
Piper
Local
PIPER_BINARY_PATH
Deepgram Aura
API
DEEPGRAM_AURA_MODEL
Avatar/3D Generation APIs#
Tavus API#
Cloud-based AI avatar video generation (optional).
Source Files#
text Copy
serwaa/services/tavus-integration/src/tavus_integration/api/client.py
serwaa/services/tavus-integration/src/tavus_integration/api/auth.py
serwaa/services/tavus-integration/src/tavus_integration/replica/manager.py
serwaa/services/tavus-integration/src/tavus_integration/persona/manager.py
Implementation Details#
Property
Value
Base URL (Prod)
https://api.tavus.io
Base URL (Sandbox)
https://sandbox.tavus.io
Auth Type
API Key (x-api-key header)
Endpoints#
Endpoint
Method
Description
/replicas
POST
Create new replica
/replicas/{id}
GET
Get replica details
/replicas/{id}/train
POST
Start training
/conversations
POST
Start conversation
/conversations/{id}
GET
Get conversation
/videos
POST
Generate video
/videos/{id}
GET
Get video status
Models#
Model
Purpose
phoenix-4
Avatar rendering
sparrow-1
Dialogue generation
raven-1
Perception/understanding
Enums#
python Copy
class ReplicaStatus(Enum):
PENDING = "pending"
TRAINING = "training"
READY = "ready"
FAILED = "failed"
DELETED = "deleted"
class ConversationStatus(Enum):
PENDING = "pending"
ACTIVE = "active"
ENDED = "ended"
FAILED = "failed"
class VideoStatus(Enum):
QUEUED = "queued"
PROCESSING = "processing"
COMPLETED = "completed"
FAILED = "failed"
Webhook Verification#
python Copy
def verify_webhook_signature(payload: bytes, signature: str, secret: str) -> bool:
"""Verify Tavus webhook signature using HMAC-SHA256."""
expected = hmac.new(
secret.encode(),
payload,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(expected, signature)
Configuration#
bash Copy
TAVUS_API_KEY=...
TAVUS_ENVIRONMENT=production # or sandbox
TAVUS_WEBHOOK_SECRET=...
Local 3D Gaussian Splatting#
Self-hosted avatar rendering engine.
Source Files#
text Copy
serwaa/services/avatar-engine/ (entire service)
Implementation Details#
Component
Description
Rendering
3D Gaussian Splatting for photorealistic avatars
Face
FLAME-based facial animation
Body
Skeleton-based body animation
Gestures
AI-driven gesture generation
Expressions
Micro-expression synthesis
GPU Requirements#
Requirement
Minimum
Recommended
GPU
NVIDIA RTX 3080 (10GB)
NVIDIA A100/H100
CUDA
11.8+
12.0+
cuDNN
8.6+
8.9+
TensorRT
8.5+
8.6+
Configuration#
bash Copy
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
AVATAR_RENDER_RESOLUTION=1080
AVATAR_RENDER_FPS=30
AVATAR_GAUSSIAN_COUNT=500000
AVATAR_USE_TENSORRT=true
Video Conferencing SDKs#
Zoom Integration#
Full Zoom meeting integration via API and browser automation.
Source Files#
text Copy
serwaa/services/video-conferencing/src/video_conferencing/zoom/api.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/auth.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/browser.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/sdk.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/adapter.py
Implementation Details#
Property
Value
REST API Base
https://api.zoom.us/v2
Auth Type
OAuth 2.0 / JWT
Browser
Playwright (headless Chromium)
REST API Endpoints#
Endpoint
Method
Description
/users/{userId}/meetings
POST
Create meeting
/users/{userId}/meetings
GET
List meetings
/meetings/{meetingId}
GET
Get meeting
/meetings/{meetingId}
PATCH
Update meeting
/meetings/{meetingId}
DELETE
Delete meeting
/meetings/{meetingId}/participants
GET
List participants
/meetings/{meetingId}/recordings
GET
Get recordings
OAuth 2.0 Flow#
python Copy
# Authorization URL
auth_url = (
f"https://zoom.us/oauth/authorize?"
f"response_type=code&client_id={client_id}&redirect_uri={redirect_uri}"
)
# Token exchange
token_response = await httpx.post(
"https://zoom.us/oauth/token",
data={
"grant_type": "authorization_code",
"code": auth_code,
"redirect_uri": redirect_uri
},
auth=(client_id, client_secret)
)
Browser Automation#
python Copy
# Join meeting via browser
async def join_meeting(meeting_url: str):
browser = await playwright.chromium.launch(headless=True)
context = await browser.new_context(
permissions=["camera", "microphone"]
)
page = await context.new_page()
await page.goto(meeting_url)
# Handle join flow...
Configuration#
bash Copy
ZOOM_CLIENT_ID=...
ZOOM_CLIENT_SECRET=...
ZOOM_WEBHOOK_SECRET=...
ZOOM_REDIRECT_URI=https://...
Microsoft Teams Integration#
Teams meeting integration via Graph API and Bot Framework.
Source Files#
text Copy
serwaa/services/video-conferencing/src/video_conferencing/teams/api.py
serwaa/services/video-conferencing/src/video_conferencing/teams/auth.py
serwaa/services/video-conferencing/src/video_conferencing/teams/browser.py
serwaa/services/video-conferencing/src/video_conferencing/teams/bot_adapter.py
Implementation Details#
Property
Value
Graph API Base
https://graph.microsoft.com/v1.0
Graph API Beta
https://graph.microsoft.com/beta
Auth Type
OAuth 2.0 (client credentials or delegated)
Bot Framework
Microsoft Bot Framework SDK
Graph API Endpoints#
Endpoint
Method
Description
/me/onlineMeetings
POST
Create meeting
/me/onlineMeetings
GET
List meetings
/communications/calls
POST
Start call
/communications/calls/{id}
GET
Get call
/communications/calls/{id}/participants
GET
List participants
/chats/{id}/messages
POST
Send message
/me/presence
GET
Get presence status
OAuth 2.0 Configuration#
python Copy
# Client credentials flow (app-only)
token_response = await httpx.post(
f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token",
data={
"grant_type": "client_credentials",
"client_id": client_id,
"client_secret": client_secret,
"scope": "https://graph.microsoft.com/.default"
}
)
# Delegated flow (on behalf of user)
auth_url = (
f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize?"
f"client_id={client_id}&response_type=code&"
f"redirect_uri={redirect_uri}&scope=Calendars.ReadWrite OnlineMeetings.ReadWrite"
)
Bot Framework Integration#
python Copy
from botbuilder.core import BotFrameworkAdapter, TurnContext
from botbuilder.schema import Activity
# Bot adapter configuration
settings = BotFrameworkAdapterSettings(
app_id=os.environ["TEAMS_BOT_APP_ID"],
app_password=os.environ["TEAMS_BOT_APP_PASSWORD"]
)
adapter = BotFrameworkAdapter(settings)
# Handle incoming activity
async def on_message(context: TurnContext):
await context.send_activity(f"You said: {context.activity.text}")
Configuration#
bash Copy
AZURE_CLIENT_ID=...
AZURE_CLIENT_SECRET=...
AZURE_TENANT_ID=...
TEAMS_BOT_APP_ID=...
TEAMS_BOT_APP_PASSWORD=...
TEAMS_BOT_CALLING_CALLBACK_URI=https://...
Google Meet Integration#
Meet integration via browser automation.
Source Files#
text Copy
serwaa/services/video-conferencing/src/video_conferencing/meet/api.py
serwaa/services/video-conferencing/src/video_conferencing/meet/auth.py
serwaa/services/video-conferencing/src/video_conferencing/meet/browser.py
serwaa/services/video-conferencing/src/video_conferencing/meet/adapter.py
Implementation Details#
Property
Value
Auth Type
OAuth 2.0
Integration
Browser automation (Playwright)
Protocol
WebRTC
OAuth 2.0 Configuration#
python Copy
# Authorization URL
auth_url = (
f"https://accounts.google.com/o/oauth2/v2/auth?"
f"client_id={client_id}&redirect_uri={redirect_uri}&"
f"response_type=code&scope=https://www.googleapis.com/auth/calendar.events"
)
Browser Automation#
python Copy
async def join_meet(meeting_code: str):
browser = await playwright.chromium.launch(headless=True)
context = await browser.new_context(
permissions=["camera", "microphone"]
)
page = await context.new_page()
await page.goto(f"https://meet.google.com/{meeting_code}")
# Handle Google sign-in and join flow...
Configuration#
bash Copy
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://...
Webex Integration#
Webex meeting integration via API.
Source Files#
text Copy
serwaa/services/video-conferencing/src/video_conferencing/webex/api.py
serwaa/services/video-conferencing/src/video_conferencing/webex/auth.py
serwaa/services/video-conferencing/src/video_conferencing/webex/adapter.py
Implementation Details#
Property
Value
API Base
https://webexapis.com/v1
Auth Type
OAuth 2.0 / Bearer Token
Endpoints#
Endpoint
Method
Description
/meetings
POST
Create meeting
/meetings/{id}
GET
Get meeting
/meetings/{id}/participants
GET
List participants
/messages
POST
Send message
Configuration#
bash Copy
WEBEX_CLIENT_ID=...
WEBEX_CLIENT_SECRET=...
WEBEX_REDIRECT_URI=https://...
WebRTC Support#
Real-time media streaming for all platforms.
Source Files#
text Copy
serwaa/services/video-conferencing/src/video_conferencing/webrtc/
Configuration#
bash Copy
# TURN/STUN servers
TURN_SERVER_URL=turn:your-turn-server.com:3478
TURN_SERVER_USERNAME=...
TURN_SERVER_PASSWORD=...
STUN_SERVER_URL=stun:stun.l.google.com:19302
Embeddings & Vector Database#
Voyage AI Embeddings#
Primary embedding provider for semantic search.
Source Files#
text Copy
serwaa/services/knowledge-base/src/knowledge_base/embedding/generators.py
serwaa/services/orchestrator/src/orchestrator/caching/embedding_service.py
Implementation Details#
Property
Value
SDK
voyageai Python package
Models
voyage-3.5-lite, voyage-3.5, voyage-3
Dimensions
1024
Usage#
python Copy
import voyageai
client = voyageai.Client(api_key=os.environ["VOYAGE_API_KEY"])
# Generate embeddings
result = client.embed(
texts=["Hello world", "How are you?"],
model="voyage-3.5-lite",
input_type="document" # or "query"
)
embeddings = result.embeddings # List of float vectors
Configuration#
bash Copy
VOYAGE_API_KEY=...
VOYAGE_MODEL=voyage-3.5-lite
Cohere Embeddings#
Fallback embedding provider.
Configuration#
bash Copy
COHERE_API_KEY=...
COHERE_MODEL=embed-english-v3.0
Qdrant Vector Database#
Self-hosted vector database for semantic search.
Implementation Details#
Property
Value
Default URL
http://localhost:6333
Protocol
REST / gRPC
Index Type
HNSW
Collection Configuration#
python Copy
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance
client = QdrantClient(url=os.environ["QDRANT_URL"])
client.create_collection(
collection_name="serwaa_knowledge",
vectors_config=VectorParams(
size=1024,
distance=Distance.COSINE
)
)
Configuration#
bash Copy
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY= # Optional
QDRANT_COLLECTION_NAME=serwaa_knowledge
Browser Automation#
Playwright#
Cross-browser automation for video conferencing.
Source Files#
text Copy
serwaa/services/computer-use/src/computer_use/actions/browser.py
serwaa/services/video-conferencing/src/video_conferencing/*/browser.py
Supported Browsers#
Browser
Headless
Use Case
Chromium
Yes
Default
Firefox
Yes
Alternative
WebKit
Yes
Safari testing
Usage Pattern#
python Copy
from playwright.async_api import async_playwright
async def browser_action():
async with async_playwright() as p:
browser = await p.chromium.launch(
headless=True,
args=["--use-fake-ui-for-media-stream"]
)
context = await browser.new_context(
permissions=["camera", "microphone", "geolocation"]
)
page = await context.new_page()
# Navigation and interaction
await page.goto("https://example.com")
await page.click("button#submit")
await page.fill("input#name", "value")
# Screenshot for Claude Computer Use
screenshot = await page.screenshot()
Claude Computer Use#
AI-driven browser/desktop automation.
Source Files#
text Copy
serwaa/services/computer-use/src/computer_use/agent/loop.py
serwaa/services/computer-use/src/computer_use/agent/recovery.py
Implementation Details#
Property
Value
LLM
Claude with tool calling
Pattern
Observe-Reason-Act loop
Max Iterations
50 (configurable)
Agent Loop#
python Copy
class ComputerUseAgent:
"""AI agent for computer automation."""
async def run(self, task: str) -> Result:
for iteration in range(self.max_iterations):
# 1. Observe: Take screenshot
screenshot = await self.take_screenshot()
# 2. Reason: Ask Claude what to do
response = await self.llm.generate(
messages=[
{"role": "user", "content": [
{"type": "text", "text": task},
{"type": "image", "source": screenshot}
]}
],
tools=self.available_tools
)
# 3. Act: Execute tool call
if response.tool_use:
result = await self.execute_tool(response.tool_use)
# 4. Verify action succeeded
if not await self.verify_action(result):
await self.recover()
# Check if task complete
if self.is_task_complete(response):
return Result.success()
Configuration#
bash Copy
FEATURE_COMPUTER_USE=true
COMPUTER_USE_MAX_ITERATIONS=50
COMPUTER_USE_ACTION_DELAY_MS=500
COMPUTER_USE_VERIFICATION_TIMEOUT_MS=5000
Mock Services#
Mock External APIs Service#
Local mock server for testing external integrations.
Source Files#
text Copy
serwaa/services/mock-external-apis/src/mock_external_apis/app.py
Mocked Endpoints#
Service
Endpoint
Description
OpenAI
/v1/embeddings
Mock embeddings
OpenAI
/v1/chat/completions
Mock chat
Anthropic
/v1/messages
Mock Claude
Deepgram
WebSocket /v1/listen
Mock STT
ElevenLabs
WebSocket /v1/text-to-speech/{id}/stream-input
Mock TTS
Cartesia
/v1/tts, /v1/tts/stream
Mock TTS
Configuration#
bash Copy
USE_MOCK_EXTERNAL_APIS=true
MOCK_EXTERNAL_APIS_BASE_URL=http://localhost:4010
MOCK_LATENCY_MS=100
# Override provider URLs to use mocks
OPENAI_BASE_URL=http://localhost:4010/v1
ANTHROPIC_BASE_URL=http://localhost:4010
DEEPGRAM_BASE_URL=http://localhost:4010
ELEVENLABS_BASE_URL=http://localhost:4010
Integration Patterns#
Provider Chain Pattern#
All major services implement a provider chain with failover:
python Copy
class ProviderChain(Generic[T]):
"""Generic provider chain with failover and routing."""
def __init__(
self,
primary: Provider[T],
fallbacks: List[Provider[T]],
max_retries: int = 2
):
self.providers = [primary] + fallbacks
self.max_retries = max_retries
async def execute(self, request: Request) -> T:
for provider in self.providers:
for attempt in range(self.max_retries):
try:
return await provider.execute(request)
except RetryableError as e:
await asyncio.sleep(2 ** attempt)
except ProviderError:
break # Try next provider
raise AllProvidersFailedError()
Error Handling#
Error Type
Action
RateLimitError
Exponential backoff, switch provider
AuthenticationError
Fail immediately, log alert
ConnectionError
Retry with backoff
ContextLengthError
Truncate context, retry
ValidationError
Fail immediately
Retry Configuration#
python Copy
class RetryConfig:
max_retries: int = 3
base_delay_ms: int = 1000
max_delay_ms: int = 30000
exponential_base: float = 2.0
jitter: bool = True
Streaming Patterns#
Protocol
Use Case
Libraries
WebSocket
Real-time bidirectional
websockets, aiohttp
SSE
Server-push events
httpx, aiohttp
gRPC
High-throughput streams
grpcio
Environment Variables Reference#
Complete Environment Variables List#
bash Copy
#=============================================================================
# AI/LLM CONFIGURATION
#=============================================================================
# Provider Chain
LLM_PRIMARY_PROVIDER=anthropic
LLM_FALLBACK_PROVIDERS=openai,google
LLM_MAX_RETRIES=2
LLM_TIMEOUT_SECONDS=60
LLM_COST_ROUTING_ENABLED=false
LLM_ENSEMBLE_ENABLED=false
LLM_RESPONSE_CACHE_ENABLED=false
# Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
ANTHROPIC_MAX_TOKENS=4096
ANTHROPIC_BASE_URL=https://api.anthropic.com
# OpenAI
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4-turbo-preview
OPENAI_BASE_URL=https://api.openai.com/v1
# Google Gemini
GOOGLE_API_KEY=AIza...
GEMINI_MODEL=gemini-1.5-pro
# Local LLM (Ollama/vLLM)
LOCAL_LLM_BASE_URL=http://localhost:11434/v1
LOCAL_LLM_MODEL=llama3.1
LOCAL_LLM_API_KEY=
#=============================================================================
# VOICE/AUDIO CONFIGURATION
#=============================================================================
# TTS Provider Chain
TTS_PRIMARY_PROVIDER=elevenlabs
TTS_FALLBACK_PROVIDERS=cartesia,openai
TTS_SAMPLE_RATE=24000
# ElevenLabs
ELEVENLABS_API_KEY=...
ELEVENLABS_MODEL=eleven_flash_v2_5
ELEVENLABS_VOICE_ID=EXAVITQu4vU6e1aXChow
# Cartesia
CARTESIA_API_KEY=...
CARTESIA_BASE_URL=https://api.cartesia.ai
# OpenAI TTS
OPENAI_TTS_MODEL=tts-1-hd
OPENAI_TTS_VOICE=alloy
# Optional TTS Providers
ORPHEUS_TTS_BASE_URL=
F5_TTS_BASE_URL=
PIPER_BINARY_PATH=
DEEPGRAM_AURA_MODEL=aura-asteria-en
# STT Provider Chain
STT_PRIMARY_PROVIDER=deepgram
STT_FALLBACK_PROVIDERS=whisper,assemblyai
STT_SAMPLE_RATE=16000
# Deepgram
DEEPGRAM_API_KEY=...
STT_MODEL=nova-2
# Whisper (Local)
WHISPER_MODEL_SIZE=medium
WHISPER_DEVICE=cuda
#=============================================================================
# AVATAR/3D CONFIGURATION
#=============================================================================
# Tavus (Optional Cloud Avatar)
TAVUS_API_KEY=...
TAVUS_ENVIRONMENT=production
TAVUS_WEBHOOK_SECRET=...
# Local Avatar Engine
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
AVATAR_RENDER_RESOLUTION=1080
AVATAR_RENDER_FPS=30
AVATAR_GAUSSIAN_COUNT=500000
AVATAR_USE_TENSORRT=true
#=============================================================================
# VIDEO CONFERENCING CONFIGURATION
#=============================================================================
# Zoom
ZOOM_CLIENT_ID=...
ZOOM_CLIENT_SECRET=...
ZOOM_WEBHOOK_SECRET=...
ZOOM_REDIRECT_URI=https://...
# Microsoft Teams
AZURE_CLIENT_ID=...
AZURE_CLIENT_SECRET=...
AZURE_TENANT_ID=...
TEAMS_BOT_APP_ID=...
TEAMS_BOT_APP_PASSWORD=...
TEAMS_BOT_CALLING_CALLBACK_URI=https://...
# Google Meet
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://...
# Webex
WEBEX_CLIENT_ID=...
WEBEX_CLIENT_SECRET=...
WEBEX_REDIRECT_URI=https://...
# WebRTC
TURN_SERVER_URL=turn:...
TURN_SERVER_USERNAME=...
TURN_SERVER_PASSWORD=...
STUN_SERVER_URL=stun:stun.l.google.com:19302
#=============================================================================
# EMBEDDINGS & VECTOR DATABASE
#=============================================================================
# Voyage AI
VOYAGE_API_KEY=...
VOYAGE_MODEL=voyage-3.5-lite
# Cohere (Fallback)
COHERE_API_KEY=...
COHERE_MODEL=embed-english-v3.0
# Qdrant
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
QDRANT_COLLECTION_NAME=serwaa_knowledge
#=============================================================================
# BROWSER AUTOMATION
#=============================================================================
# Computer Use
FEATURE_COMPUTER_USE=true
COMPUTER_USE_MAX_ITERATIONS=50
COMPUTER_USE_ACTION_DELAY_MS=500
COMPUTER_USE_VERIFICATION_TIMEOUT_MS=5000
COMPUTER_USE_CONFIRMATION_TIMEOUT_MS=30000
#=============================================================================
# MOCK SERVICES (Testing)
#=============================================================================
USE_MOCK_EXTERNAL_APIS=false
MOCK_EXTERNAL_APIS_BASE_URL=http://localhost:4010
MOCK_LATENCY_MS=100
Migration Considerations#
High Priority Integrations#
These integrations are critical for core functionality:
Integration
Priority
Complexity
Notes
Anthropic Claude
P0
Low
Primary LLM, well-documented
Deepgram STT
P0
Medium
WebSocket complexity
ElevenLabs TTS
P0
Medium
WebSocket streaming
Qdrant
P0
Low
Standard REST API
Local Avatar Engine
P0
High
GPU requirements
Medium Priority Integrations#
Integration
Priority
Complexity
Notes
OpenAI
P1
Low
Fallback LLM
Voyage AI
P1
Low
Embeddings
Zoom
P1
Medium
OAuth complexity
Teams
P1
High
Bot Framework
Lower Priority Integrations#
Integration
Priority
Complexity
Notes
Google Gemini
P2
Low
Alternative LLM
Cartesia
P2
Low
Alternative TTS
Tavus
P2
Medium
Optional cloud avatar
Meet/Webex
P2
Medium
Additional platforms
Migration Strategy#
Phase 1: Core AI - Migrate LLM provider chain with Anthropic/OpenAI
Phase 2: Voice - Migrate STT/TTS with Deepgram/ElevenLabs
Phase 3: Avatar - Port local 3D Gaussian Splatting engine
Phase 4: Conferencing - Implement Zoom/Teams integrations
Phase 5: Extensions - Add remaining providers and features
TypeScript Migration Notes#
When migrating Python integrations to TypeScript:
Python Library
TypeScript Equivalent
anthropic
@anthropic-ai/sdk
openai
openai
httpx
node-fetch, axios, undici
websockets
ws, socket.io-client
voyageai
Custom client (REST)
qdrant-client
@qdrant/js-client-rest
playwright
playwright
Authentication Handling#
All OAuth 2.0 flows should use:
Secure token storage (encrypted at rest)
Automatic token refresh
Proper scope management
PKCE for public clients
Appendix: API Response Examples#
Anthropic Claude Response#
json Copy
{
"id": "msg_01XFDUDYJgAACzvnptvVoYEL",
"type": "message",
"role": "assistant",
"content": [
{
"type": "text",
"text": "Hello! How can I help you today?"
}
],
"model": "claude-3-5-sonnet-20241022",
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 25,
"output_tokens": 12
}
}
Deepgram Transcription Response#
json Copy
{
"type": "Results",
"channel_index": [0, 1],
"duration": 2.5,
"start": 0.0,
"is_final": true,
"speech_final": true,
"channel": {
"alternatives": [
{
"transcript": "Hello, how are you today?",
"confidence": 0.98,
"words": [
{
"word": "Hello",
"start": 0.0,
"end": 0.4,
"confidence": 0.99,
"punctuated_word": "Hello,"
},
{
"word": "how",
"start": 0.5,
"end": 0.7,
"confidence": 0.98,
"punctuated_word": "how"
},
{
"word": "are",
"start": 0.8,
"end": 0.9,
"confidence": 0.97,
"punctuated_word": "are"
},
{
"word": "you",
"start": 1.0,
"end": 1.2,
"confidence": 0.99,
"punctuated_word": "you"
},
{
"word": "today",
"start": 1.3,
"end": 1.8,
"confidence": 0.96,
"punctuated_word": "today?"
}
]
}
]
}
}
ElevenLabs TTS Response#
json Copy
{
"audio": "base64_encoded_pcm_audio_data...",
"isFinal": false,
"normalizedAlignment": {
"charStartTimesMs": [0, 100, 200],
"charDurationsMs": [100, 100, 150]
}
}
Voyage AI Embedding Response#
json Copy
{
"object": "list",
"data": [
{
"object": "embedding",
"embedding": [0.0234, -0.0156, 0.0089, ...],
"index": 0
}
],
"model": "voyage-3.5-lite",
"usage": {
"total_tokens": 15
}
}