Disciplines · Integrations

ElevenLabs Integration Documentation

ElevenLabs provides high-quality text-to-speech capabilities including:

11sections2 minread
Markdown source
On this page

This directory contains comprehensive documentation for integrating ElevenLabs text-to-speech services into the Oshun platform.

Document Description
API Reference Complete API documentation with types and examples
Integration Guide Step-by-step integration guide and best practices
Troubleshooting Common issues and solutions
Migration Guide Migrate from other TTS providers

Overview#

ElevenLabs provides high-quality text-to-speech capabilities including:

  • Text-to-Speech: Convert text to natural-sounding speech
  • Voice Cloning: Create custom voices from audio samples
  • Streaming: Real-time audio streaming for low-latency applications
  • Voice Library: Access to pre-made and community voices
  • Speech-to-Speech: Voice conversion with emotion preservation
  • Sound Effects: AI-generated sound effects
  • Audio Isolation: Remove background noise from audio

Architecture#

text
┌─────────────────────────────────────────────────────────────────────────────┐
│                     Oshun ElevenLabs Integration                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  Application Layer                                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  Lilith Consciousness    │    Yemaya Creative    │    Isis Factory  │    │
│  │  - Conversational AI     │    - Voice Design     │    - Batch TTS   │    │
│  │  - Streaming responses   │    - Voice cloning    │    - Dubbing     │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                      │                                       │
│  Service Layer                       ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    ElevenLabs Provider                               │    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│  │  │ TTS Provider │  │ Voice Clone  │  │ STS Provider │              │    │
│  │  │              │  │ Provider     │  │              │              │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│  │                                                                      │    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│  │  │ Voice Lib    │  │ Sound FX     │  │ Audio        │              │    │
│  │  │ Provider     │  │ Provider     │  │ Isolation    │              │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                      │                                       │
│  Infrastructure                      ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│  │  │ Rate Limiter │  │ Circuit      │  │ Retry        │              │    │
│  │  │              │  │ Breaker      │  │ Handler      │              │    │
│  │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                      │                                       │
│                                      ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                     ElevenLabs API                                   │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Getting Started#

Prerequisites#

  1. ElevenLabs account with API key
  2. Node.js 18+ environment
  3. Network access to ElevenLabs API

Quick Start#

typescript
import { ElevenLabsProvider } from '@oshun/elevenlabs-provider';

// Initialize provider
const elevenlabs = new ElevenLabsProvider({
  apiKey: process.env.ELEVENLABS_API_KEY,
});

// Text-to-speech
const audio = await elevenlabs.textToSpeech({
  voiceId: 'Rachel',
  text: 'Hello, world!',
  modelId: 'eleven_multilingual_v2',
});

// Streaming
const stream = elevenlabs.textToSpeechStream({
  voiceId: 'Rachel',
  text: 'Hello, world!',
});

for await (const chunk of stream) {
  // Process audio chunk
}

Available Models#

Model ID Use Case Languages Latency Quality
eleven_multilingual_v2 General purpose, highest quality 29 Medium Excellent
eleven_turbo_v2_5 Low latency, real-time apps 32 Low Very Good
eleven_monolingual_v1 English only, legacy 1 Medium Good
eleven_english_sts_v2 Speech-to-speech 1 Medium Very Good

Voice Selection#

Pre-made Voices#

ElevenLabs provides a variety of pre-made voices:

Voice Gender Age Accent Use Case
Rachel Female Young Adult American Narration
Domi Female Young Adult American Conversation
Bella Female Young Adult American Soft, gentle
Antoni Male Young Adult American Warm, friendly
Adam Male Middle Aged American Deep, narration
Arnold Male Middle Aged American Crisp, formal

Voice Cloning#

Create custom voices from audio samples:

typescript
const customVoice = await elevenlabs.cloneVoice({
  name: 'My Custom Voice',
  files: [audioSample1, audioSample2],
  description: 'Professional narrator voice',
});

Best Practices#

  1. Choose the right model: Use Turbo for real-time, Multilingual for quality
  2. Tune voice settings: Adjust stability and similarity for best results
  3. Handle rate limits: Implement exponential backoff
  4. Cache audio: Store generated audio to avoid redundant API calls
  5. Monitor usage: Track character consumption against quotas
  6. Use streaming: For long content, stream audio progressively

Error Handling#

The integration includes comprehensive error handling:

typescript
try {
  const audio = await elevenlabs.textToSpeech({...});
} catch (error) {
  if (error instanceof RateLimitError) {
    // Handle rate limiting
  } else if (error instanceof QuotaExceededError) {
    // Handle quota issues
  } else if (error instanceof VoiceNotFoundError) {
    // Handle missing voice
  }
}

See Troubleshooting for detailed error resolution.

Monitoring#

Key metrics to monitor:

  • Character usage: Track against subscription quota
  • API latency: Monitor response times
  • Error rates: Track failures by error type
  • Cache hit rate: Optimize caching strategy

Support#

  • Voice Analytics — ElevenLabs capability surfaced through the shared Isis audio providers (libs/isis/ai-providers); no dedicated per-feature module
  • Pronunciation Dictionaries — applied via the shared Isis TTS provider (libs/isis/ai-providers, tts family)
  • Audio Enhancement — applied via the shared Isis audio providers (libs/isis/ai-providers)
  • Dubbing Studio — ElevenLabs dubbing surfaced through the shared Isis providers (libs/isis/ai-providers, conversational-ai/tts)