Eve library

Psyche Domain: External API Integrations

Guide

Revision 1

On this page

This document provides a comprehensive inventory of all external API integrations used by the Serwaa platform (source codebase for the Psyche domain migration).

Table of Contents#

  1. Overview
  2. AI/LLM APIs
  3. Voice/Audio APIs
  4. Avatar/3D Generation APIs
  5. Video Conferencing SDKs
  6. Embeddings & Vector Database
  7. Browser Automation
  8. Mock Services
  9. Integration Patterns
  10. Environment Variables Reference
  11. Migration Considerations

Overview#

The Serwaa platform integrates with 50+ external APIs across 8 major categories:

Category Primary Provider Fallback Providers Integration Type
LLM/AI Anthropic Claude OpenAI, Google Gemini, Local LLM REST, Streaming
Text-to-Speech ElevenLabs Cartesia, OpenAI, Deepgram Aura WebSocket, REST
Speech-to-Text Deepgram Whisper, AssemblyAI WebSocket, REST
Avatar Rendering 3D Gaussian Splatting Tavus (optional) Local, REST
Video Conferencing Multi-platform - SDK, Browser, WebRTC
Embeddings Voyage AI Cohere, OpenAI REST
Vector Database Qdrant - REST
Browser Automation Playwright - Library

Integration Architecture#

text
┌─────────────────────────────────────────────────────────────────┐
│                      Provider Chain Pattern                      │
├─────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   Primary    │→ │   Fallback   │→ │   Fallback   │          │
│  │   Provider   │  │   Provider   │  │   Provider   │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│        ↓                  ↓                  ↓                  │
│  ┌────────────────────────────────────────────────────┐        │
│  │              Unified Response Interface             │        │
│  └────────────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────────┘

AI/LLM APIs#

Anthropic Claude API#

Primary LLM provider for conversation and reasoning.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
serwaa/services/knowledge-base/src/knowledge_base/rag/generator.py
serwaa/services/computer-use/src/computer_use/agent/loop.py

Implementation Details#

Property Value
Base URL https://api.anthropic.com (configurable)
API Version 2023-06-01
Endpoint POST /v1/messages
Auth Type API Key (x-api-key header)
SDK anthropic Python package

Supported Models#

Model ID Context Window Use Case
claude-3-5-sonnet-20241022 200K tokens Default, balanced
claude-3-opus-20240229 200K tokens Complex reasoning
claude-3-sonnet-20240229 200K tokens Balanced
claude-3-haiku-20240307 200K tokens Fast, cost-effective

Features#

  • Streaming Responses: Server-sent events via messages.stream()
  • Tool Calling: Function definitions via tools parameter
  • System Prompts: Persona and behavior configuration
  • Token Counting: Native token counting for context management
  • Vision: Image understanding in messages (for Computer Use)

Request Schema#

python
{
    "model": "claude-3-5-sonnet-20241022",
    "max_tokens": 4096,
    "system": "You are a helpful assistant...",
    "messages": [
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hi there!"},
        {"role": "user", "content": "What is the weather?"}
    ],
    "tools": [
        {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "input_schema": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                },
                "required": ["location"]
            }
        }
    ],
    "stream": True
}

Response Schema (Streaming)#

python
# Message start event
{"type": "message_start", "message": {"id": "msg_...", "role": "assistant"}}

# Content delta events
{"type": "content_block_delta", "delta": {"type": "text_delta", "text": "Hello"}}

# Tool use events
{"type": "content_block_start", "content_block": {"type": "tool_use", "name": "get_weather"}}

# Message end event
{"type": "message_stop"}

Error Handling#

Error Type HTTP Code Handling
RateLimitError 429 Exponential backoff, switch provider
InvalidRequestError 400 Context truncation if too long
APIConnectionError - Retry with backoff
AuthenticationError 401 Fail immediately

Configuration#

bash
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
ANTHROPIC_MAX_TOKENS=4096
ANTHROPIC_BASE_URL=https://api.anthropic.com

OpenAI GPT API#

Fallback LLM provider and TTS/embeddings source.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py
serwaa/services/voice-engine/src/voice_engine/tts/openai_tts.py

Implementation Details#

Property Value
Base URL https://api.openai.com/v1 (configurable)
Endpoint (Chat) POST /chat/completions
Endpoint (TTS) POST /audio/speech
Endpoint (Embeddings) POST /embeddings
Auth Type Bearer Token
SDK openai Python package

Supported Models#

Model ID Context Window Use Case
gpt-4-turbo-preview 128K tokens Default fallback
gpt-4 8K tokens Legacy
gpt-3.5-turbo 16K tokens Cost-effective

Text-to-Speech#

Property Value
Model tts-1-hd
Voices alloy, echo, fable, onyx, nova, shimmer
Output Format MP3, AAC, FLAC, WAV
Sample Rate 16000-24000 Hz

Features#

  • Streaming: SSE via stream=True
  • Tool Calling: Function definitions
  • JSON Mode: Structured output via response_format
  • Token Counting: Via tiktoken library

Configuration#

bash
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4-turbo-preview
OPENAI_BASE_URL=https://api.openai.com/v1

Google Gemini API#

Alternative LLM provider with multimodal capabilities.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py

Implementation Details#

Property Value
Base URL https://generativelanguage.googleapis.com/v1beta
Endpoint (Generate) POST /models/{model}:generateContent
Endpoint (Stream) POST /models/{model}:streamGenerateContent
Endpoint (Count) POST /models/{model}:countTokens
Auth Type API Key (query parameter)

Supported Models#

Model ID Context Window Use Case
gemini-1.5-pro 1M tokens Long context
gemini-1.5-flash 1M tokens Fast inference
gemini-pro 32K tokens Standard

Features#

  • System Instructions: Via systemInstruction field
  • Function Calling: Via tools.functionDeclarations
  • Streaming: Server-sent events
  • Token Counting: Native endpoint

Configuration#

bash
GOOGLE_API_KEY=AIza...
GEMINI_MODEL=gemini-1.5-pro

Local LLM (Ollama/vLLM/llama.cpp)#

Self-hosted LLM for offline or cost-sensitive deployments.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py

Implementation Details#

Property Value
Base URL Configurable (default: http://localhost:11434/v1)
Endpoint POST /chat/completions (OpenAI-compatible)
Auth Type Optional Bearer Token
Servers Ollama, vLLM, llama.cpp

Supported Models#

Model Parameters Use Case
llama3.1 8B-70B General purpose
mistral 7B Fast inference
mixtral 8x7B MoE model

Configuration#

bash
LOCAL_LLM_BASE_URL=http://localhost:11434/v1
LOCAL_LLM_MODEL=llama3.1
LOCAL_LLM_API_KEY=  # Optional

LLM Provider Chain#

Orchestrates multiple LLM providers with failover and routing.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/conversation/llm.py

Class: LLMProviderChain#

python
class LLMProviderChain:
    """Manages multiple LLM providers with failover and routing."""

    def __init__(
        self,
        primary_provider: str = "anthropic",
        fallback_providers: List[str] = ["openai", "google"],
        max_retries: int = 2,
        cost_routing_enabled: bool = False,
        ensemble_enabled: bool = False,
        cache_enabled: bool = False
    ):
        ...

    async def generate(
        self,
        messages: List[Message],
        tools: Optional[List[Tool]] = None,
        **kwargs
    ) -> Response:
        """Generate with automatic failover."""
        ...

    async def generate_stream(
        self,
        messages: List[Message],
        **kwargs
    ) -> AsyncIterator[StreamEvent]:
        """Stream generation with automatic failover."""
        ...

Features#

Feature Description Environment Variable
Failover Switch providers on failure LLM_FALLBACK_PROVIDERS
Cost Routing Route based on cost/token LLM_COST_ROUTING_ENABLED
Ensemble Run multiple in parallel LLM_ENSEMBLE_ENABLED
Caching TTL-based response cache LLM_RESPONSE_CACHE_ENABLED
Retry Exponential backoff LLM_MAX_RETRIES

Configuration#

bash
LLM_PRIMARY_PROVIDER=anthropic
LLM_FALLBACK_PROVIDERS=openai,google
LLM_MAX_RETRIES=2
LLM_TIMEOUT_SECONDS=60
LLM_COST_ROUTING_ENABLED=false
LLM_ENSEMBLE_ENABLED=false
LLM_RESPONSE_CACHE_ENABLED=false

Voice/Audio APIs#

ElevenLabs TTS#

Primary text-to-speech provider with ultra-low latency streaming.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/tts/elevenlabs.py

Implementation Details#

Property Value
WebSocket URL wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input
HTTP Base URL https://api.elevenlabs.io/v1
Auth Type API Key (xi-api-key header)
Latency ~75ms first byte

Models#

Model ID Latency Quality Use Case
eleven_flash_v2_5 Ultra-low Good Real-time
eleven_turbo_v2_5 Low Better Balanced
eleven_multilingual_v2 Medium Best Multilingual

Voice Configuration#

python
{
    "voice_id": "EXAVITQu4vU6e1aXChow",  # Bella (default)
    "model_id": "eleven_flash_v2_5",
    "voice_settings": {
        "stability": 0.5,
        "similarity_boost": 0.75,
        "style": 0.0,
        "use_speaker_boost": True
    },
    "output_format": "pcm_24000"  # PCM 24kHz
}

Output Formats#

Format Sample Rate Use Case
pcm_16000 16 kHz Low bandwidth
pcm_22050 22.05 kHz Standard
pcm_24000 24 kHz High quality
pcm_44100 44.1 kHz Maximum quality
mp3_44100_128 44.1 kHz Compressed

WebSocket Protocol#

python
# Connection
ws = await websocket.connect(
    f"wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}/stream-input"
    f"?model_id={model_id}&output_format={output_format}"
)

# Send text chunks
await ws.send(json.dumps({
    "text": "Hello ",
    "voice_settings": {...},
    "xi_api_key": api_key
}))

# Receive audio chunks
async for message in ws:
    data = json.loads(message)
    if "audio" in data:
        audio_bytes = base64.b64decode(data["audio"])
        yield audio_bytes

# End stream
await ws.send(json.dumps({"text": ""}))

Configuration#

bash
ELEVENLABS_API_KEY=...
ELEVENLABS_MODEL=eleven_flash_v2_5
ELEVENLABS_VOICE_ID=EXAVITQu4vU6e1aXChow

Cartesia Sonic TTS#

Alternative low-latency TTS provider.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/tts/cartesia.py

Implementation Details#

Property Value
Base URL https://api.cartesia.ai
Endpoint (Batch) POST /v1/tts
Endpoint (Stream) POST /v1/tts/stream
Auth Type Bearer Token

Request Schema#

python
{
    "model": "sonic",
    "text": "Hello world",
    "voice_id": "...",
    "output_format": "pcm_16bit",
    "sample_rate": 24000,
    "language": "en"
}

Features#

  • Streaming: Chunked audio response
  • Voice Cloning: Custom voice support
  • Multi-language: English, Spanish, French, etc.

Configuration#

bash
CARTESIA_API_KEY=...
CARTESIA_BASE_URL=https://api.cartesia.ai

Deepgram STT#

Primary speech-to-text provider with real-time WebSocket streaming.

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/stt/deepgram.py

Implementation Details#

Property Value
WebSocket URL wss://api.deepgram.com/v1/listen
Auth Type Token (Authorization: Token {api_key})
Latency ~100-300ms

Query Parameters#

Parameter Default Description
model nova-2 Transcription model
encoding linear16 Audio encoding
sample_rate 16000 Sample rate in Hz
channels 1 Audio channels
punctuate true Add punctuation
smart_format true Smart formatting
interim_results true Send partial results
utterance_end_ms 1000 Silence threshold
diarize false Speaker identification
language en Language code

Models#

Model Latency Accuracy Use Case
nova-2 Low Best Default
nova Low Good Fallback
enhanced Medium Better Non-realtime
base Low Basic Cost-effective

WebSocket Protocol#

python
# Connection
ws = await websocket.connect(
    f"wss://api.deepgram.com/v1/listen?"
    f"model=nova-2&encoding=linear16&sample_rate=16000&"
    f"punctuate=true&interim_results=true"
)
ws.extra_headers = {"Authorization": f"Token {api_key}"}

# Send audio chunks
await ws.send(audio_bytes)

# Receive transcription
async for message in ws:
    data = json.loads(message)
    if data["type"] == "Results":
        transcript = data["channel"]["alternatives"][0]["transcript"]
        is_final = data["is_final"]

        # Word-level details
        for word in data["channel"]["alternatives"][0]["words"]:
            text = word["word"]
            start = word["start"]
            end = word["end"]
            confidence = word["confidence"]
            speaker = word.get("speaker")  # If diarization enabled

# Keep-alive (every 10 seconds)
await ws.send(json.dumps({"type": "KeepAlive"}))

# Close stream
await ws.send(json.dumps({"type": "CloseStream"}))

Response Schema#

python
{
    "type": "Results",
    "channel_index": [0, 1],
    "duration": 1.5,
    "start": 0.0,
    "is_final": True,
    "speech_final": True,
    "channel": {
        "alternatives": [
            {
                "transcript": "Hello world",
                "confidence": 0.98,
                "words": [
                    {"word": "Hello", "start": 0.0, "end": 0.5, "confidence": 0.99},
                    {"word": "world", "start": 0.6, "end": 1.0, "confidence": 0.97}
                ]
            }
        ]
    }
}

Configuration#

bash
DEEPGRAM_API_KEY=...
STT_MODEL=nova-2
STT_SAMPLE_RATE=16000

OpenAI Whisper STT#

Fallback speech-to-text (local or API-based).

Source Files#

text
serwaa/services/voice-engine/src/voice_engine/stt/whisper.py

Implementation Details#

Property Value
API Endpoint POST /v1/audio/transcriptions
Local openai-whisper Python package
Auth Type Bearer Token (API)

Models (Local)#

Model Parameters VRAM Speed
tiny 39M ~1GB Fastest
base 74M ~1GB Fast
small 244M ~2GB Balanced
medium 769M ~5GB Good
large 1550M ~10GB Best

Configuration#

bash
# API mode
OPENAI_API_KEY=...
WHISPER_MODEL=whisper-1

# Local mode
WHISPER_MODEL_SIZE=medium
WHISPER_DEVICE=cuda  # or cpu

Voice Provider Chain#

Orchestrates TTS and STT providers with failover.

Configuration#

bash
# TTS Configuration
TTS_PRIMARY_PROVIDER=elevenlabs
TTS_FALLBACK_PROVIDERS=cartesia,openai
TTS_SAMPLE_RATE=24000

# STT Configuration
STT_PRIMARY_PROVIDER=deepgram
STT_FALLBACK_PROVIDERS=whisper,assemblyai
STT_SAMPLE_RATE=16000

Additional TTS Providers#

Provider Type Environment Variables
Orpheus Self-hosted ORPHEUS_TTS_BASE_URL
F5-TTS Voice cloning F5_TTS_BASE_URL
Piper Local PIPER_BINARY_PATH
Deepgram Aura API DEEPGRAM_AURA_MODEL

Avatar/3D Generation APIs#

Tavus API#

Cloud-based AI avatar video generation (optional).

Source Files#

text
serwaa/services/tavus-integration/src/tavus_integration/api/client.py
serwaa/services/tavus-integration/src/tavus_integration/api/auth.py
serwaa/services/tavus-integration/src/tavus_integration/replica/manager.py
serwaa/services/tavus-integration/src/tavus_integration/persona/manager.py

Implementation Details#

Property Value
Base URL (Prod) https://api.tavus.io
Base URL (Sandbox) https://sandbox.tavus.io
Auth Type API Key (x-api-key header)

Endpoints#

Endpoint Method Description
/replicas POST Create new replica
/replicas/{id} GET Get replica details
/replicas/{id}/train POST Start training
/conversations POST Start conversation
/conversations/{id} GET Get conversation
/videos POST Generate video
/videos/{id} GET Get video status

Models#

Model Purpose
phoenix-4 Avatar rendering
sparrow-1 Dialogue generation
raven-1 Perception/understanding

Enums#

python
class ReplicaStatus(Enum):
    PENDING = "pending"
    TRAINING = "training"
    READY = "ready"
    FAILED = "failed"
    DELETED = "deleted"

class ConversationStatus(Enum):
    PENDING = "pending"
    ACTIVE = "active"
    ENDED = "ended"
    FAILED = "failed"

class VideoStatus(Enum):
    QUEUED = "queued"
    PROCESSING = "processing"
    COMPLETED = "completed"
    FAILED = "failed"

Webhook Verification#

python
def verify_webhook_signature(payload: bytes, signature: str, secret: str) -> bool:
    """Verify Tavus webhook signature using HMAC-SHA256."""
    expected = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

Configuration#

bash
TAVUS_API_KEY=...
TAVUS_ENVIRONMENT=production  # or sandbox
TAVUS_WEBHOOK_SECRET=...

Local 3D Gaussian Splatting#

Self-hosted avatar rendering engine.

Source Files#

text
serwaa/services/avatar-engine/  (entire service)

Implementation Details#

Component Description
Rendering 3D Gaussian Splatting for photorealistic avatars
Face FLAME-based facial animation
Body Skeleton-based body animation
Gestures AI-driven gesture generation
Expressions Micro-expression synthesis

GPU Requirements#

Requirement Minimum Recommended
GPU NVIDIA RTX 3080 (10GB) NVIDIA A100/H100
CUDA 11.8+ 12.0+
cuDNN 8.6+ 8.9+
TensorRT 8.5+ 8.6+

Configuration#

bash
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
AVATAR_RENDER_RESOLUTION=1080
AVATAR_RENDER_FPS=30
AVATAR_GAUSSIAN_COUNT=500000
AVATAR_USE_TENSORRT=true

Video Conferencing SDKs#

Zoom Integration#

Full Zoom meeting integration via API and browser automation.

Source Files#

text
serwaa/services/video-conferencing/src/video_conferencing/zoom/api.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/auth.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/browser.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/sdk.py
serwaa/services/video-conferencing/src/video_conferencing/zoom/adapter.py

Implementation Details#

Property Value
REST API Base https://api.zoom.us/v2
Auth Type OAuth 2.0 / JWT
Browser Playwright (headless Chromium)

REST API Endpoints#

Endpoint Method Description
/users/{userId}/meetings POST Create meeting
/users/{userId}/meetings GET List meetings
/meetings/{meetingId} GET Get meeting
/meetings/{meetingId} PATCH Update meeting
/meetings/{meetingId} DELETE Delete meeting
/meetings/{meetingId}/participants GET List participants
/meetings/{meetingId}/recordings GET Get recordings

OAuth 2.0 Flow#

python
# Authorization URL
auth_url = (
    f"https://zoom.us/oauth/authorize?"
    f"response_type=code&client_id={client_id}&redirect_uri={redirect_uri}"
)

# Token exchange
token_response = await httpx.post(
    "https://zoom.us/oauth/token",
    data={
        "grant_type": "authorization_code",
        "code": auth_code,
        "redirect_uri": redirect_uri
    },
    auth=(client_id, client_secret)
)

Browser Automation#

python
# Join meeting via browser
async def join_meeting(meeting_url: str):
    browser = await playwright.chromium.launch(headless=True)
    context = await browser.new_context(
        permissions=["camera", "microphone"]
    )
    page = await context.new_page()
    await page.goto(meeting_url)
    # Handle join flow...

Configuration#

bash
ZOOM_CLIENT_ID=...
ZOOM_CLIENT_SECRET=...
ZOOM_WEBHOOK_SECRET=...
ZOOM_REDIRECT_URI=https://...

Microsoft Teams Integration#

Teams meeting integration via Graph API and Bot Framework.

Source Files#

text
serwaa/services/video-conferencing/src/video_conferencing/teams/api.py
serwaa/services/video-conferencing/src/video_conferencing/teams/auth.py
serwaa/services/video-conferencing/src/video_conferencing/teams/browser.py
serwaa/services/video-conferencing/src/video_conferencing/teams/bot_adapter.py

Implementation Details#

Property Value
Graph API Base https://graph.microsoft.com/v1.0
Graph API Beta https://graph.microsoft.com/beta
Auth Type OAuth 2.0 (client credentials or delegated)
Bot Framework Microsoft Bot Framework SDK

Graph API Endpoints#

Endpoint Method Description
/me/onlineMeetings POST Create meeting
/me/onlineMeetings GET List meetings
/communications/calls POST Start call
/communications/calls/{id} GET Get call
/communications/calls/{id}/participants GET List participants
/chats/{id}/messages POST Send message
/me/presence GET Get presence status

OAuth 2.0 Configuration#

python
# Client credentials flow (app-only)
token_response = await httpx.post(
    f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token",
    data={
        "grant_type": "client_credentials",
        "client_id": client_id,
        "client_secret": client_secret,
        "scope": "https://graph.microsoft.com/.default"
    }
)

# Delegated flow (on behalf of user)
auth_url = (
    f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize?"
    f"client_id={client_id}&response_type=code&"
    f"redirect_uri={redirect_uri}&scope=Calendars.ReadWrite OnlineMeetings.ReadWrite"
)

Bot Framework Integration#

python
from botbuilder.core import BotFrameworkAdapter, TurnContext
from botbuilder.schema import Activity

# Bot adapter configuration
settings = BotFrameworkAdapterSettings(
    app_id=os.environ["TEAMS_BOT_APP_ID"],
    app_password=os.environ["TEAMS_BOT_APP_PASSWORD"]
)
adapter = BotFrameworkAdapter(settings)

# Handle incoming activity
async def on_message(context: TurnContext):
    await context.send_activity(f"You said: {context.activity.text}")

Configuration#

bash
AZURE_CLIENT_ID=...
AZURE_CLIENT_SECRET=...
AZURE_TENANT_ID=...
TEAMS_BOT_APP_ID=...
TEAMS_BOT_APP_PASSWORD=...
TEAMS_BOT_CALLING_CALLBACK_URI=https://...

Google Meet Integration#

Meet integration via browser automation.

Source Files#

text
serwaa/services/video-conferencing/src/video_conferencing/meet/api.py
serwaa/services/video-conferencing/src/video_conferencing/meet/auth.py
serwaa/services/video-conferencing/src/video_conferencing/meet/browser.py
serwaa/services/video-conferencing/src/video_conferencing/meet/adapter.py

Implementation Details#

Property Value
Auth Type OAuth 2.0
Integration Browser automation (Playwright)
Protocol WebRTC

OAuth 2.0 Configuration#

python
# Authorization URL
auth_url = (
    f"https://accounts.google.com/o/oauth2/v2/auth?"
    f"client_id={client_id}&redirect_uri={redirect_uri}&"
    f"response_type=code&scope=https://www.googleapis.com/auth/calendar.events"
)

Browser Automation#

python
async def join_meet(meeting_code: str):
    browser = await playwright.chromium.launch(headless=True)
    context = await browser.new_context(
        permissions=["camera", "microphone"]
    )
    page = await context.new_page()
    await page.goto(f"https://meet.google.com/{meeting_code}")
    # Handle Google sign-in and join flow...

Configuration#

bash
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://...

Webex Integration#

Webex meeting integration via API.

Source Files#

text
serwaa/services/video-conferencing/src/video_conferencing/webex/api.py
serwaa/services/video-conferencing/src/video_conferencing/webex/auth.py
serwaa/services/video-conferencing/src/video_conferencing/webex/adapter.py

Implementation Details#

Property Value
API Base https://webexapis.com/v1
Auth Type OAuth 2.0 / Bearer Token

Endpoints#

Endpoint Method Description
/meetings POST Create meeting
/meetings/{id} GET Get meeting
/meetings/{id}/participants GET List participants
/messages POST Send message

Configuration#

bash
WEBEX_CLIENT_ID=...
WEBEX_CLIENT_SECRET=...
WEBEX_REDIRECT_URI=https://...

WebRTC Support#

Real-time media streaming for all platforms.

Source Files#

text
serwaa/services/video-conferencing/src/video_conferencing/webrtc/

Configuration#

bash
# TURN/STUN servers
TURN_SERVER_URL=turn:your-turn-server.com:3478
TURN_SERVER_USERNAME=...
TURN_SERVER_PASSWORD=...
STUN_SERVER_URL=stun:stun.l.google.com:19302

Embeddings & Vector Database#

Voyage AI Embeddings#

Primary embedding provider for semantic search.

Source Files#

text
serwaa/services/knowledge-base/src/knowledge_base/embedding/generators.py
serwaa/services/orchestrator/src/orchestrator/caching/embedding_service.py

Implementation Details#

Property Value
SDK voyageai Python package
Models voyage-3.5-lite, voyage-3.5, voyage-3
Dimensions 1024

Usage#

python
import voyageai

client = voyageai.Client(api_key=os.environ["VOYAGE_API_KEY"])

# Generate embeddings
result = client.embed(
    texts=["Hello world", "How are you?"],
    model="voyage-3.5-lite",
    input_type="document"  # or "query"
)
embeddings = result.embeddings  # List of float vectors

Configuration#

bash
VOYAGE_API_KEY=...
VOYAGE_MODEL=voyage-3.5-lite

Cohere Embeddings#

Fallback embedding provider.

Configuration#

bash
COHERE_API_KEY=...
COHERE_MODEL=embed-english-v3.0

Qdrant Vector Database#

Self-hosted vector database for semantic search.

Implementation Details#

Property Value
Default URL http://localhost:6333
Protocol REST / gRPC
Index Type HNSW

Collection Configuration#

python
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance

client = QdrantClient(url=os.environ["QDRANT_URL"])

client.create_collection(
    collection_name="serwaa_knowledge",
    vectors_config=VectorParams(
        size=1024,
        distance=Distance.COSINE
    )
)

Configuration#

bash
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=  # Optional
QDRANT_COLLECTION_NAME=serwaa_knowledge

Browser Automation#

Playwright#

Cross-browser automation for video conferencing.

Source Files#

text
serwaa/services/computer-use/src/computer_use/actions/browser.py
serwaa/services/video-conferencing/src/video_conferencing/*/browser.py

Supported Browsers#

Browser Headless Use Case
Chromium Yes Default
Firefox Yes Alternative
WebKit Yes Safari testing

Usage Pattern#

python
from playwright.async_api import async_playwright

async def browser_action():
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--use-fake-ui-for-media-stream"]
        )
        context = await browser.new_context(
            permissions=["camera", "microphone", "geolocation"]
        )
        page = await context.new_page()

        # Navigation and interaction
        await page.goto("https://example.com")
        await page.click("button#submit")
        await page.fill("input#name", "value")

        # Screenshot for Claude Computer Use
        screenshot = await page.screenshot()

Claude Computer Use#

AI-driven browser/desktop automation.

Source Files#

text
serwaa/services/computer-use/src/computer_use/agent/loop.py
serwaa/services/computer-use/src/computer_use/agent/recovery.py

Implementation Details#

Property Value
LLM Claude with tool calling
Pattern Observe-Reason-Act loop
Max Iterations 50 (configurable)

Agent Loop#

python
class ComputerUseAgent:
    """AI agent for computer automation."""

    async def run(self, task: str) -> Result:
        for iteration in range(self.max_iterations):
            # 1. Observe: Take screenshot
            screenshot = await self.take_screenshot()

            # 2. Reason: Ask Claude what to do
            response = await self.llm.generate(
                messages=[
                    {"role": "user", "content": [
                        {"type": "text", "text": task},
                        {"type": "image", "source": screenshot}
                    ]}
                ],
                tools=self.available_tools
            )

            # 3. Act: Execute tool call
            if response.tool_use:
                result = await self.execute_tool(response.tool_use)

                # 4. Verify action succeeded
                if not await self.verify_action(result):
                    await self.recover()

            # Check if task complete
            if self.is_task_complete(response):
                return Result.success()

Configuration#

bash
FEATURE_COMPUTER_USE=true
COMPUTER_USE_MAX_ITERATIONS=50
COMPUTER_USE_ACTION_DELAY_MS=500
COMPUTER_USE_VERIFICATION_TIMEOUT_MS=5000

Mock Services#

Mock External APIs Service#

Local mock server for testing external integrations.

Source Files#

text
serwaa/services/mock-external-apis/src/mock_external_apis/app.py

Mocked Endpoints#

Service Endpoint Description
OpenAI /v1/embeddings Mock embeddings
OpenAI /v1/chat/completions Mock chat
Anthropic /v1/messages Mock Claude
Deepgram WebSocket /v1/listen Mock STT
ElevenLabs WebSocket /v1/text-to-speech/{id}/stream-input Mock TTS
Cartesia /v1/tts, /v1/tts/stream Mock TTS

Configuration#

bash
USE_MOCK_EXTERNAL_APIS=true
MOCK_EXTERNAL_APIS_BASE_URL=http://localhost:4010
MOCK_LATENCY_MS=100

# Override provider URLs to use mocks
OPENAI_BASE_URL=http://localhost:4010/v1
ANTHROPIC_BASE_URL=http://localhost:4010
DEEPGRAM_BASE_URL=http://localhost:4010
ELEVENLABS_BASE_URL=http://localhost:4010

Integration Patterns#

Provider Chain Pattern#

All major services implement a provider chain with failover:

python
class ProviderChain(Generic[T]):
    """Generic provider chain with failover and routing."""

    def __init__(
        self,
        primary: Provider[T],
        fallbacks: List[Provider[T]],
        max_retries: int = 2
    ):
        self.providers = [primary] + fallbacks
        self.max_retries = max_retries

    async def execute(self, request: Request) -> T:
        for provider in self.providers:
            for attempt in range(self.max_retries):
                try:
                    return await provider.execute(request)
                except RetryableError as e:
                    await asyncio.sleep(2 ** attempt)
                except ProviderError:
                    break  # Try next provider
        raise AllProvidersFailedError()

Error Handling#

Error Type Action
RateLimitError Exponential backoff, switch provider
AuthenticationError Fail immediately, log alert
ConnectionError Retry with backoff
ContextLengthError Truncate context, retry
ValidationError Fail immediately

Retry Configuration#

python
class RetryConfig:
    max_retries: int = 3
    base_delay_ms: int = 1000
    max_delay_ms: int = 30000
    exponential_base: float = 2.0
    jitter: bool = True

Streaming Patterns#

Protocol Use Case Libraries
WebSocket Real-time bidirectional websockets, aiohttp
SSE Server-push events httpx, aiohttp
gRPC High-throughput streams grpcio

Environment Variables Reference#

Complete Environment Variables List#

bash
#=============================================================================
# AI/LLM CONFIGURATION
#=============================================================================

# Provider Chain
LLM_PRIMARY_PROVIDER=anthropic
LLM_FALLBACK_PROVIDERS=openai,google
LLM_MAX_RETRIES=2
LLM_TIMEOUT_SECONDS=60
LLM_COST_ROUTING_ENABLED=false
LLM_ENSEMBLE_ENABLED=false
LLM_RESPONSE_CACHE_ENABLED=false

# Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-3-5-sonnet-20241022
ANTHROPIC_MAX_TOKENS=4096
ANTHROPIC_BASE_URL=https://api.anthropic.com

# OpenAI
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4-turbo-preview
OPENAI_BASE_URL=https://api.openai.com/v1

# Google Gemini
GOOGLE_API_KEY=AIza...
GEMINI_MODEL=gemini-1.5-pro

# Local LLM (Ollama/vLLM)
LOCAL_LLM_BASE_URL=http://localhost:11434/v1
LOCAL_LLM_MODEL=llama3.1
LOCAL_LLM_API_KEY=

#=============================================================================
# VOICE/AUDIO CONFIGURATION
#=============================================================================

# TTS Provider Chain
TTS_PRIMARY_PROVIDER=elevenlabs
TTS_FALLBACK_PROVIDERS=cartesia,openai
TTS_SAMPLE_RATE=24000

# ElevenLabs
ELEVENLABS_API_KEY=...
ELEVENLABS_MODEL=eleven_flash_v2_5
ELEVENLABS_VOICE_ID=EXAVITQu4vU6e1aXChow

# Cartesia
CARTESIA_API_KEY=...
CARTESIA_BASE_URL=https://api.cartesia.ai

# OpenAI TTS
OPENAI_TTS_MODEL=tts-1-hd
OPENAI_TTS_VOICE=alloy

# Optional TTS Providers
ORPHEUS_TTS_BASE_URL=
F5_TTS_BASE_URL=
PIPER_BINARY_PATH=
DEEPGRAM_AURA_MODEL=aura-asteria-en

# STT Provider Chain
STT_PRIMARY_PROVIDER=deepgram
STT_FALLBACK_PROVIDERS=whisper,assemblyai
STT_SAMPLE_RATE=16000

# Deepgram
DEEPGRAM_API_KEY=...
STT_MODEL=nova-2

# Whisper (Local)
WHISPER_MODEL_SIZE=medium
WHISPER_DEVICE=cuda

#=============================================================================
# AVATAR/3D CONFIGURATION
#=============================================================================

# Tavus (Optional Cloud Avatar)
TAVUS_API_KEY=...
TAVUS_ENVIRONMENT=production
TAVUS_WEBHOOK_SECRET=...

# Local Avatar Engine
AVATAR_MODEL_PATH=/models/avatar
AVATAR_CHECKPOINT_PATH=/models/avatar/checkpoints
AVATAR_RENDER_RESOLUTION=1080
AVATAR_RENDER_FPS=30
AVATAR_GAUSSIAN_COUNT=500000
AVATAR_USE_TENSORRT=true

#=============================================================================
# VIDEO CONFERENCING CONFIGURATION
#=============================================================================

# Zoom
ZOOM_CLIENT_ID=...
ZOOM_CLIENT_SECRET=...
ZOOM_WEBHOOK_SECRET=...
ZOOM_REDIRECT_URI=https://...

# Microsoft Teams
AZURE_CLIENT_ID=...
AZURE_CLIENT_SECRET=...
AZURE_TENANT_ID=...
TEAMS_BOT_APP_ID=...
TEAMS_BOT_APP_PASSWORD=...
TEAMS_BOT_CALLING_CALLBACK_URI=https://...

# Google Meet
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://...

# Webex
WEBEX_CLIENT_ID=...
WEBEX_CLIENT_SECRET=...
WEBEX_REDIRECT_URI=https://...

# WebRTC
TURN_SERVER_URL=turn:...
TURN_SERVER_USERNAME=...
TURN_SERVER_PASSWORD=...
STUN_SERVER_URL=stun:stun.l.google.com:19302

#=============================================================================
# EMBEDDINGS & VECTOR DATABASE
#=============================================================================

# Voyage AI
VOYAGE_API_KEY=...
VOYAGE_MODEL=voyage-3.5-lite

# Cohere (Fallback)
COHERE_API_KEY=...
COHERE_MODEL=embed-english-v3.0

# Qdrant
QDRANT_URL=http://localhost:6333
QDRANT_API_KEY=
QDRANT_COLLECTION_NAME=serwaa_knowledge

#=============================================================================
# BROWSER AUTOMATION
#=============================================================================

# Computer Use
FEATURE_COMPUTER_USE=true
COMPUTER_USE_MAX_ITERATIONS=50
COMPUTER_USE_ACTION_DELAY_MS=500
COMPUTER_USE_VERIFICATION_TIMEOUT_MS=5000
COMPUTER_USE_CONFIRMATION_TIMEOUT_MS=30000

#=============================================================================
# MOCK SERVICES (Testing)
#=============================================================================

USE_MOCK_EXTERNAL_APIS=false
MOCK_EXTERNAL_APIS_BASE_URL=http://localhost:4010
MOCK_LATENCY_MS=100

Migration Considerations#

High Priority Integrations#

These integrations are critical for core functionality:

Integration Priority Complexity Notes
Anthropic Claude P0 Low Primary LLM, well-documented
Deepgram STT P0 Medium WebSocket complexity
ElevenLabs TTS P0 Medium WebSocket streaming
Qdrant P0 Low Standard REST API
Local Avatar Engine P0 High GPU requirements

Medium Priority Integrations#

Integration Priority Complexity Notes
OpenAI P1 Low Fallback LLM
Voyage AI P1 Low Embeddings
Zoom P1 Medium OAuth complexity
Teams P1 High Bot Framework

Lower Priority Integrations#

Integration Priority Complexity Notes
Google Gemini P2 Low Alternative LLM
Cartesia P2 Low Alternative TTS
Tavus P2 Medium Optional cloud avatar
Meet/Webex P2 Medium Additional platforms

Migration Strategy#

  1. Phase 1: Core AI - Migrate LLM provider chain with Anthropic/OpenAI
  2. Phase 2: Voice - Migrate STT/TTS with Deepgram/ElevenLabs
  3. Phase 3: Avatar - Port local 3D Gaussian Splatting engine
  4. Phase 4: Conferencing - Implement Zoom/Teams integrations
  5. Phase 5: Extensions - Add remaining providers and features

TypeScript Migration Notes#

When migrating Python integrations to TypeScript:

Python Library TypeScript Equivalent
anthropic @anthropic-ai/sdk
openai openai
httpx node-fetch, axios, undici
websockets ws, socket.io-client
voyageai Custom client (REST)
qdrant-client @qdrant/js-client-rest
playwright playwright

Authentication Handling#

All OAuth 2.0 flows should use:

  • Secure token storage (encrypted at rest)
  • Automatic token refresh
  • Proper scope management
  • PKCE for public clients

Appendix: API Response Examples#

Anthropic Claude Response#

json
{
  "id": "msg_01XFDUDYJgAACzvnptvVoYEL",
  "type": "message",
  "role": "assistant",
  "content": [
    {
      "type": "text",
      "text": "Hello! How can I help you today?"
    }
  ],
  "model": "claude-3-5-sonnet-20241022",
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": {
    "input_tokens": 25,
    "output_tokens": 12
  }
}

Deepgram Transcription Response#

json
{
  "type": "Results",
  "channel_index": [0, 1],
  "duration": 2.5,
  "start": 0.0,
  "is_final": true,
  "speech_final": true,
  "channel": {
    "alternatives": [
      {
        "transcript": "Hello, how are you today?",
        "confidence": 0.98,
        "words": [
          {
            "word": "Hello",
            "start": 0.0,
            "end": 0.4,
            "confidence": 0.99,
            "punctuated_word": "Hello,"
          },
          {
            "word": "how",
            "start": 0.5,
            "end": 0.7,
            "confidence": 0.98,
            "punctuated_word": "how"
          },
          {
            "word": "are",
            "start": 0.8,
            "end": 0.9,
            "confidence": 0.97,
            "punctuated_word": "are"
          },
          {
            "word": "you",
            "start": 1.0,
            "end": 1.2,
            "confidence": 0.99,
            "punctuated_word": "you"
          },
          {
            "word": "today",
            "start": 1.3,
            "end": 1.8,
            "confidence": 0.96,
            "punctuated_word": "today?"
          }
        ]
      }
    ]
  }
}

ElevenLabs TTS Response#

json
{
  "audio": "base64_encoded_pcm_audio_data...",
  "isFinal": false,
  "normalizedAlignment": {
    "charStartTimesMs": [0, 100, 200],
    "charDurationsMs": [100, 100, 150]
  }
}

Voyage AI Embedding Response#

json
{
  "object": "list",
  "data": [
    {
      "object": "embedding",
      "embedding": [0.0234, -0.0156, 0.0089, ...],
      "index": 0
    }
  ],
  "model": "voyage-3.5-lite",
  "usage": {
    "total_tokens": 15
  }
}