Eve library

ElevenLabs Integration

Guide

Revision 1

On this page

Reference documentation for the ElevenLabs AI voice provider supporting text-to-speech, streaming, voice cloning, and audio tags.

Overview#

The ElevenLabs provider offers:

  • Text-to-Speech: Multiple models (Flash, Turbo, Multilingual, v3)
  • Streaming Audio: Real-time audio generation
  • Voice Management: Listing, preview, cloning
  • Audio Tags (v3): Performance cues for emotional delivery
  • Model Listing: Available models and capabilities
  • Retry Logic: Exponential backoff with rate limit handling

Configuration#

Environment Variables#

Variable Description Required
ELEVENLABS_API_KEY ElevenLabs API key Yes

Provider Initialization#

typescript
import { ElevenLabsProvider } from './providers/elevenlabs-provider';

const provider = new ElevenLabsProvider({
  apiKey: process.env.ELEVENLABS_API_KEY,
  baseURL: 'https://api.elevenlabs.io/v1',
  maxRetries: 3,
  retryDelayMs: 1000,
  timeout: 30000,
  defaultModel: 'eleven_multilingual_v2',
  defaultVoiceId: '21m00Tcm4TlvDq8ikWAM',
  defaultOutputFormat: 'mp3_44100_128',
});

Configuration Options#

Option Type Default Description
apiKey string ELEVENLABS_API_KEY API key
baseURL string https://api.elevenlabs.io/v1 API base URL
maxRetries number 3 Max retry attempts
retryDelayMs number 1000 Base retry delay (ms)
timeout number 30000 Request timeout (ms)
defaultModel string eleven_multilingual_v2 Default TTS model
defaultVoiceId string - Default voice ID
defaultOutputFormat OutputFormat mp3_44100_128 Default audio format

Text-to-Speech#

Basic TTS#

typescript
const response = await provider.textToSpeech({
  text: 'Welcome to your guided meditation.',
  voiceId: 'pNInz6obpgDQGcFmaJgB', // Adam
  modelId: 'eleven_multilingual_v2',
});

// response.audio is a Buffer containing the audio
fs.writeFileSync('output.mp3', response.audio);

TTS Request Parameters#

Parameter Type Description
text string Text to convert to speech
voiceId string Voice ID to use
modelId string Model ID (optional)
voiceSettings VoiceSettings Voice customization
outputFormat OutputFormat Audio format
languageCode string Language override
seed number For reproducible output
previousText string Context from previous segment
nextText string Context for next segment
optimizeStreamingLatency 0-4 Latency optimization level

Voice Settings#

typescript
const response = await provider.textToSpeech({
  text: 'This is a calm, slow meditation.',
  voiceId: 'voice_id',
  voiceSettings: {
    stability: 0.8, // 0-1, higher = more consistent
    similarity_boost: 0.75, // 0-1, higher = closer to original
    style: 0.3, // 0-1, v2+ models only
    use_speaker_boost: true, // Enhance voice clarity
    speed: 0.8, // 0.25-4.0, speech rate
  },
});

Output Formats#

Format Description
mp3_22050_32 MP3, 22.05kHz, 32kbps
mp3_44100_32 MP3, 44.1kHz, 32kbps
mp3_44100_64 MP3, 44.1kHz, 64kbps
mp3_44100_96 MP3, 44.1kHz, 96kbps
mp3_44100_128 MP3, 44.1kHz, 128kbps (default)
mp3_44100_192 MP3, 44.1kHz, 192kbps
pcm_16000 PCM, 16kHz
pcm_22050 PCM, 22.05kHz
pcm_24000 PCM, 24kHz
pcm_44100 PCM, 44.1kHz
ulaw_8000 uLaw, 8kHz (telephony)
alaw_8000 aLaw, 8kHz (telephony)
opus_48000_64 Opus, 48kHz, 64kbps
opus_48000_128 Opus, 48kHz, 128kbps

TTS Response#

typescript
interface TTSResponse {
  audio: Buffer;
  contentType: string;
  requestId?: string;
  characterCount?: number;
  historyItemId?: string;
}

Streaming Audio#

Stream TTS#

typescript
const stream = await provider.textToSpeechStream({
  text: 'This is a long meditation script...',
  voiceId: 'voice_id',
});

for await (const chunk of stream) {
  // chunk.audio is a Buffer
  // chunk.isFinal indicates last chunk
  // chunk.alignment contains timing info

  socket.write(chunk.audio);

  if (chunk.alignment) {
    console.log('Characters:', chunk.alignment.chars);
    console.log('Timings:', chunk.alignment.charStartTimesMs);
  }
}

Stream Chunk#

typescript
interface TTSStreamChunk {
  audio: Buffer;
  isFinal: boolean;
  alignment?: {
    chars: string[];
    charStartTimesMs: number[];
    charDurationsMs: number[];
  };
}

Latency Optimization#

typescript
// Level 0: No optimization (highest quality)
// Level 1: Slight latency reduction
// Level 2: Balanced
// Level 3: Optimized for real-time
// Level 4: Maximum speed (lowest quality)

const response = await provider.textToSpeech({
  text: 'Hello!',
  voiceId: 'voice_id',
  optimizeStreamingLatency: 3,
});

Voice Management#

List Voices#

typescript
const response = await provider.listVoices({
  page_size: 20,
  search: 'calm',
  sort: 'name',
  sort_direction: 'asc',
  voice_type: 'default',
  category: 'premade',
});

for (const voice of response.voices) {
  console.log(`${voice.name} (${voice.voice_id})`);
  console.log(`  Category: ${voice.category}`);
  console.log(`  Labels: ${JSON.stringify(voice.labels)}`);
  console.log(`  Preview: ${voice.preview_url}`);
}

Voice Properties#

typescript
interface Voice {
  voice_id: string;
  name: string;
  category: 'premade' | 'cloned' | 'generated' | 'professional';
  labels?: Record<string, string>;
  description?: string;
  preview_url?: string;
  settings?: VoiceSettings;
  samples?: VoiceSample[];
  high_quality_base_model_ids?: string[];
}

Voice Search Parameters#

Parameter Type Description
page_size number Results per page
search string Search query
sort string Sort field (created_at, name)
sort_direction string asc or desc
voice_type string personal, community, default, workspace
category string premade, cloned, generated, professional

Get Voice Sample#

typescript
const sampleUrl = await provider.getVoiceSample('voice_id');
// Returns URL to audio sample

Voice Cloning#

Comprehensive voice cloning support with Instant Voice Cloning (IVC), Professional Voice Cloning (PVC), consent management, and quality analysis.

Supported Audio Formats#

Format MIME Type Extension
MP3 audio/mpeg .mp3
WAV audio/wav, audio/x-wav .wav
M4A audio/mp4, audio/x-m4a .m4a
OGG audio/ogg .ogg
FLAC audio/flac .flac
WebM audio/webm .webm

Instant Voice Cloning (IVC)#

typescript
const response = await provider.cloneVoice({
  name: 'My Custom Voice',
  files: [
    { data: audioBuffer1, filename: 'sample1.mp3' },
    { data: audioBuffer2, filename: 'sample2.mp3' },
  ],
  description: 'A calm, meditative voice',
  labels: { use_case: 'meditation', accent: 'american' },
  removeBackgroundNoise: true,
});

console.log('Created voice:', response.voice_id);

Voice Cloning Request#

typescript
interface VoiceCloningRequest {
  name: string; // Voice name (required)
  files: VoiceCloningSample[]; // Audio samples (required)
  description?: string; // Voice description
  labels?: Record<string, string>; // Categorization labels
  removeBackgroundNoise?: boolean; // Remove noise from samples
}

interface VoiceCloningSample {
  data: Buffer; // Audio data
  filename: string; // Used for MIME type detection
  mimeType?: string; // Override MIME type
}

Voice Cloning Response#

typescript
interface VoiceCloningResponse {
  voice_id: string; // Unique voice identifier
}

Edit Voice Metadata#

typescript
await provider.editVoice('voice_id', {
  name: 'Updated Name',
  description: 'New description',
  labels: { style: 'calming' },
});

Update Voice Settings#

typescript
await provider.updateVoiceSettings('voice_id', {
  stability: 0.8,
  similarity_boost: 0.7,
  style: 0.3,
  use_speaker_boost: true,
});

Add Samples to Voice#

typescript
await provider.addVoiceSamples('voice_id', {
  files: [{ data: newAudioBuffer, filename: 'new_sample.mp3' }],
  removeBackgroundNoise: true,
});

Delete Voice Sample#

typescript
await provider.deleteVoiceSample('voice_id', 'sample_id');

Delete Voice#

typescript
await provider.deleteVoice('voice_id');

Professional Voice Cloning (PVC)#

Higher quality cloning with manual verification (requires 30+ minutes of audio):

typescript
const response = await provider.professionalVoiceClone({
  name: 'Professional Voice',
  files: [{ data: highQualityAudio, filename: 'studio_recording.wav' }],
  description: 'Professional narrator voice',
  consentVerificationId: 'consent_id_from_flow',
});

// Check status
if (response.status === 'processing') {
  console.log('Estimated completion:', response.estimated_completion);
}

PVC Status Values#

Status Description
pending_verification Awaiting consent verification
processing Voice is being trained
completed Voice ready for use
failed Training failed
rejected Consent rejected

Check Cloning Capabilities#

typescript
// Check if IVC is available
const canUseIVC = await provider.canUseInstantVoiceCloning();

// Check if PVC is available
const canUsePVC = await provider.canUseProfessionalVoiceCloning();

// Get detailed limits
const limits = await provider.getVoiceCloningLimits();
// {
//   voiceLimit: 10,
//   currentVoiceCount: 3,
//   maxVoiceAddEdits: 100,
//   voiceAddEditCounter: 15,
//   canUseIVC: true,
//   canUsePVC: false
// }

Consent flow for ethical voice cloning with verification and tamper detection.

typescript
const consent = provider.generateConsentStatement({
  voiceOwnerName: 'John Smith',
  clonedVoiceName: 'Meditation Guide',
  intendedUse: 'Meditation and wellness applications',
  expirationDate: new Date('2026-01-01'),
});

console.log(consent.consentText);
console.log(consent.checksum); // For verification
typescript
interface ConsentStatement {
  version: string; // Statement format version
  timestamp: string; // When consent was given
  voiceOwnerName: string; // Voice owner's name
  clonedVoiceName: string; // Name for cloned voice
  intendedUse: string; // Description of use
  expirationDate?: string; // Optional expiration
  consentText: string; // Full consent agreement
  checksum?: string; // Tamper verification
}
typescript
const result = provider.verifyConsentStatement(consent);

if (result.valid) {
  console.log('Consent is valid');
} else {
  console.log('Errors:', result.errors);
  if (result.isExpired) {
    console.log('Consent has expired');
  }
}
typescript
interface ConsentVerificationResult {
  valid: boolean; // Overall validity
  errors: string[]; // Validation errors
  isExpired: boolean; // Expiration status
}

Sample Validation#

Utility functions for validating audio samples before cloning.

Validate Single Sample#

typescript
import { validateVoiceCloningSample } from './providers/elevenlabs-provider';

const sample = { data: audioBuffer, filename: 'sample.mp3' };
const result = validateVoiceCloningSample(sample);

if (!result.valid) {
  console.log('Validation errors:', result.errors);
}

Validation Rules#

Check Requirement
Data Non-empty buffer
Filename Required, non-empty
Format Supported MIME type
Min Size 10KB minimum
Max Size 50MB maximum

Analyze Sample Quality#

typescript
import { analyzeVoiceCloningSamples } from './providers/elevenlabs-provider';

const samples = [
  { data: buffer1, filename: 'sample1.wav' },
  { data: buffer2, filename: 'sample2.wav' },
];

const report = analyzeVoiceCloningSamples(samples);

console.log('Quality Score:', report.qualityScore);
console.log('Recommendations:', report.recommendations);
console.log('Meets Requirements:', report.meetsMinimumRequirements);
console.log('Details:', report.details);

Quality Report Structure#

typescript
interface VoiceCloningQualityReport {
  qualityScore: number; // 0-100 score
  recommendations: string[]; // Improvement suggestions
  meetsMinimumRequirements: boolean; // Minimum met
  details: {
    totalDuration: number; // Estimated seconds
    sampleCount: number; // Number of samples
    averageSampleSize: number; // Bytes
    formatConsistency: boolean; // Same format
  };
}

Best Practices#

typescript
import { VOICE_CLONING_BEST_PRACTICES } from './providers/elevenlabs-provider';

// {
//   minSampleDuration: 60,      // 60 seconds minimum
//   optimalSampleDuration: 300, // 5 minutes optimal
//   maxSampleDuration: 1800,    // 30 minutes max
//   recommendedSampleCount: 3,  // 3 samples
//   recommendedSampleRate: 44100,
//   recommendedBitDepth: 16,
//   tips: [...]
// }

Recording Tips#

Tip Description
Environment Quiet room, minimal background noise
Microphone Good quality, consistent distance
Speaking Natural pace and tone
Content Diverse text, varied expressions
Format WAV or FLAC for best quality
Audio Mono, not stereo
Volume Consistent levels across samples

MIME Type Detection#

typescript
import { getMimeTypeFromFilename } from './providers/elevenlabs-provider';

const mimeType = getMimeTypeFromFilename('sample.wav');
// 'audio/wav'

const mimeType2 = getMimeTypeFromFilename('audio.m4a');
// 'audio/mp4'

Complete Voice Cloning Workflow#

typescript
import {
  validateVoiceCloningSample,
  analyzeVoiceCloningSamples,
  VOICE_CLONING_BEST_PRACTICES,
} from './providers/elevenlabs-provider';

// 1. Validate individual samples
const samples = [
  { data: buffer1, filename: 'sample1.wav' },
  { data: buffer2, filename: 'sample2.wav' },
];

for (const sample of samples) {
  const validation = validateVoiceCloningSample(sample);
  if (!validation.valid) {
    throw new Error(`Invalid sample: ${validation.errors.join(', ')}`);
  }
}

// 2. Analyze overall quality
const report = analyzeVoiceCloningSamples(samples);
if (!report.meetsMinimumRequirements) {
  console.warn('Recommendations:', report.recommendations);
}

// 3. Generate consent
const consent = provider.generateConsentStatement({
  voiceOwnerName: 'Jane Doe',
  clonedVoiceName: 'Wellness Guide',
  intendedUse: 'Meditation app narration',
});

// 4. Verify consent
const verification = provider.verifyConsentStatement(consent);
if (!verification.valid) {
  throw new Error(`Invalid consent: ${verification.errors.join(', ')}`);
}

// 5. Check account capabilities
const canClone = await provider.canUseInstantVoiceCloning();
if (!canClone) {
  throw new Error('Voice cloning not available on this account');
}

// 6. Clone the voice
const response = await provider.cloneVoice({
  name: consent.clonedVoiceName,
  files: samples,
  description: 'Created with proper consent',
  labels: { consent_verified: 'true' },
  removeBackgroundNoise: true,
});

console.log('Voice created:', response.voice_id);

Voice Design#

Create unique AI voices from text descriptions without audio samples.

Design a Voice#

typescript
const response = await provider.designVoice({
  voiceDescription:
    'A calm, soothing voice ideal for meditation. Warm tone with natural pauses.',
  text: 'Welcome to your guided meditation session. Take a deep breath in...',
  guidanceScale: 6,
  autoGenerateText: false,
});

// Returns 3 previews to choose from
for (const preview of response.previews) {
  console.log('Voice ID:', preview.generatedVoiceId);
  console.log('Duration:', preview.durationSecs, 'seconds');
  // preview.audioBase64 contains the audio data
}

Voice Design Request#

typescript
interface VoiceDesignRequest {
  voiceDescription: string; // 20-1000 characters (required)
  text?: string; // 100-1000 character preview text
  modelId?: VoiceDesignModelId; // Model to use
  autoGenerateText?: boolean; // Auto-generate preview text
  guidanceScale?: number; // 0-10, how closely AI follows prompt
  loudness?: number; // -1 to 1, volume level
  seed?: number; // For reproducibility
  shouldEnhance?: boolean; // AI enhancement of description
  quality?: number; // Output quality vs. variety
  referenceAudioBase64?: string; // Reference audio (v3 only)
  promptStrength?: number; // 0-1, prompt vs. reference balance
  outputFormat?: OutputFormat; // Audio format
}

Voice Design Model IDs#

Model Description
eleven_multilingual_ttv_v2 Multilingual voice design (v2)
eleven_ttv_v3 Latest voice design model (v3)

Voice Design Response#

typescript
interface VoiceDesignResponse {
  previews: VoiceDesignPreview[]; // Typically 3 previews
  text: string; // Text used for previews
}

interface VoiceDesignPreview {
  audioBase64: string; // Base64-encoded audio
  generatedVoiceId: string; // ID for saving
  mediaType: string; // Audio MIME type
  durationSecs: number; // Audio duration
  language?: string; // Detected language
}

Save Designed Voice#

typescript
const savedVoice = await provider.saveDesignedVoice({
  voiceName: 'Meditation Guide',
  voiceDescription: 'A calm, soothing voice for meditation',
  generatedVoiceId: response.previews[0].generatedVoiceId,
  labels: { category: 'meditation', mood: 'calm' },
  playedNotSelectedVoiceIds: [
    // Feedback for ElevenLabs
    response.previews[1].generatedVoiceId,
    response.previews[2].generatedVoiceId,
  ],
});

console.log('Saved voice:', savedVoice.voice_id);

Voice Save Request#

typescript
interface VoiceSaveRequest {
  voiceName: string; // Name (required)
  voiceDescription: string; // Description
  generatedVoiceId: string; // From preview (required)
  labels?: Record<string, string>; // Metadata
  playedNotSelectedVoiceIds?: string[]; // Feedback
}

Voice Save Response#

typescript
interface VoiceSaveResponse {
  voice_id: string;
  name: string;
  category: 'generated' | 'cloned' | 'premade' | 'professional';
  description?: string;
  settings?: VoiceSettings;
  labels?: Record<string, string>;
  created_at_unix?: number;
}

Convenience Methods#

Create Voice from Description#

Design and save in one call:

typescript
const voice = await provider.createVoiceFromDescription({
  voiceDescription: 'A wise, grounded voice for spiritual teachings',
  voiceName: 'Spiritual Teacher',
  previewText: 'The journey inward is the most important journey...',
  guidanceScale: 7,
  labels: { category: 'spiritual', style: 'wise' },
});

Create Voice from Preset#

Use built-in presets:

typescript
const voice = await provider.createVoiceFromPreset(
  'Meditation Guide',
  'My Meditation Voice',
  {
    labels: { app: 'meditation-app' },
    previewText: 'Custom preview text...',
  }
);

Voice Design Presets#

Pre-configured presets for common use cases.

Available Presets#

Preset Category Description
Meditation Guide meditation Calm, soothing, nurturing
Spiritual Teacher spiritual Wise, resonant, grounded
Yoga Instructor meditation Balanced, encouraging, peaceful
Sleep Narrator meditation Soft, gentle, dreamlike
Philosophical Narrator educational Thoughtful, articulate
Mystical Storyteller storytelling Enchanting, otherworldly

Design from Preset#

typescript
const response = await provider.designVoiceFromPreset('Meditation Guide', {
  text: 'Custom preview text instead of preset sample...',
  guidanceScale: 7, // Override preset guidance
});

Preset Structure#

typescript
interface VoiceDesignPreset {
  name: string;
  description: string;
  category:
    | 'meditation'
    | 'spiritual'
    | 'educational'
    | 'storytelling'
    | 'character'
    | 'custom';
  voiceDescription: string;
  guidanceScale: number;
  recommendedSettings: Partial<VoiceSettings>;
  sampleTexts: string[];
}

Get Preset by Name#

typescript
import { getVoiceDesignPreset } from './providers/elevenlabs-provider';

const preset = getVoiceDesignPreset('Meditation Guide');
console.log(preset?.voiceDescription);
console.log(preset?.sampleTexts[0]);

Get Presets by Category#

typescript
import { getVoiceDesignPresetsByCategory } from './providers/elevenlabs-provider';

const meditationPresets = getVoiceDesignPresetsByCategory('meditation');
// ['Meditation Guide', 'Yoga Instructor', 'Sleep Narrator']

Create Custom Preset#

typescript
import { createVoiceDesignPreset } from './providers/elevenlabs-provider';

const customPreset = createVoiceDesignPreset(
  'Zen Master',
  'A profound, deeply peaceful voice that speaks with the wisdom of ages...',
  {
    category: 'spiritual',
    description: 'Ancient wisdom voice',
    guidanceScale: 8,
    recommendedSettings: {
      stability: 0.75,
      similarity_boost: 0.65,
      style: 0.4,
      speed: 0.85,
    },
    sampleTexts: [
      'In the silence between thoughts, truth reveals itself.',
      'The path to enlightenment begins with a single breath.',
    ],
  }
);

Built-in Preset Details#

Meditation Guide:

typescript
{
  voiceDescription: 'A calm, serene voice with a gentle, soothing quality...',
  guidanceScale: 6,
  recommendedSettings: {
    stability: 0.7,
    similarity_boost: 0.6,
    style: 0.3,
    speed: 0.85
  }
}

Spiritual Teacher:

typescript
{
  voiceDescription: 'A wise, deeply resonant voice that conveys ancient wisdom...',
  guidanceScale: 7,
  recommendedSettings: {
    stability: 0.65,
    similarity_boost: 0.7,
    style: 0.4,
    speed: 0.9
  }
}

Sleep Narrator:

typescript
{
  voiceDescription: 'An extremely soft, gentle voice with a dreamlike quality...',
  guidanceScale: 8,
  recommendedSettings: {
    stability: 0.8,
    similarity_boost: 0.5,
    style: 0.2,
    speed: 0.75
  }
}

Complete Voice Design Workflow#

typescript
import {
  VOICE_DESIGN_PRESETS,
  getVoiceDesignPreset,
  getVoiceDesignPresetsByCategory,
  createVoiceDesignPreset,
} from './providers/elevenlabs-provider';

// 1. List available presets
console.log(
  'Available presets:',
  VOICE_DESIGN_PRESETS.map((p) => p.name)
);

// 2. Filter by category
const spiritualPresets = getVoiceDesignPresetsByCategory('spiritual');

// 3. Get a specific preset
const preset = getVoiceDesignPreset('Meditation Guide')!;

// 4. Generate voice previews
const designResponse = await provider.designVoiceFromPreset(
  'Meditation Guide',
  {
    text: preset.sampleTexts[0],
  }
);

// 5. Review previews (play audio to user)
for (const preview of designResponse.previews) {
  const audioBuffer = Buffer.from(preview.audioBase64, 'base64');
  // Play or analyze audio...
}

// 6. Save the best preview
const selectedPreview = designResponse.previews[0];
const voice = await provider.saveDesignedVoice({
  voiceName: 'My Meditation Guide',
  voiceDescription: preset.voiceDescription,
  generatedVoiceId: selectedPreview.generatedVoiceId,
  labels: {
    category: preset.category,
    source: 'voice_design',
  },
  playedNotSelectedVoiceIds: designResponse.previews
    .filter((p) => p.generatedVoiceId !== selectedPreview.generatedVoiceId)
    .map((p) => p.generatedVoiceId),
});

// 7. Apply recommended settings
await provider.updateVoiceSettings(voice.voice_id, preset.recommendedSettings);

// 8. Use the voice for TTS
const audio = await provider.textToSpeech({
  text: 'Welcome to your meditation...',
  voiceId: voice.voice_id,
});

Model Management#

List Models#

typescript
const models = await provider.listModels();

for (const model of models) {
  console.log(`${model.name} (${model.model_id})`);
  console.log(`  TTS: ${model.can_do_text_to_speech}`);
  console.log(`  Voice Conversion: ${model.can_do_voice_conversion}`);
  console.log(`  Style: ${model.can_use_style}`);
  console.log(`  Languages: ${model.languages?.map((l) => l.name).join(', ')}`);
}

Available Models#

Model ID Description
eleven_monolingual_v1 English only, original
eleven_multilingual_v1 Multi-language support
eleven_multilingual_v2 Improved multi-language
eleven_turbo_v2 Fast generation
eleven_turbo_v2_5 Latest turbo model
eleven_flash_v2 Ultra-fast generation
eleven_flash_v2_5 Latest flash model

Usage Stats#

typescript
const stats = await provider.getUsageStats();

console.log(
  `Characters used: ${stats.character_count}/${stats.character_limit}`
);
console.log(`Voices: ${stats.voice_add_edit_counter}/${stats.voice_limit}`);
console.log(`Reset: ${new Date(stats.next_character_count_reset_unix * 1000)}`);

Audio Tags (v3)#

Audio tags control emotional delivery in the v3 model.

Tag Categories#

Category Description Examples
emotion Emotional states [sad], [happy], [calm]
delivery Speech delivery [whispers], [shouts]
reaction Vocal reactions [sighs], [laughs], [gasps]
cognitive Thinking patterns [pauses], [hesitates]
sound_effect Non-speech sounds [applause], [thunder]
accent Speaking styles [French accent], [pirate voice]
pacing Speed control [slowly], [quickly]
intensity Strength modifiers [softly], [loudly]
character Character voices [old man voice], [narrator]

Emotion Tags#

typescript
const emotionTags = [
  '[sad]',
  '[happy]',
  '[angry]',
  '[excited]',
  '[nervous]',
  '[calm]',
  '[sorrowful]',
  '[hopeful]',
  '[fearful]',
  '[surprised]',
  '[tender]',
  '[loving]',
  '[contemplative]',
  '[serene]',
  '[determined]',
];

Delivery Tags#

typescript
const deliveryTags = [
  '[whispers]',
  '[shouts]',
  '[murmurs]',
  '[speaks softly]',
  '[speaks firmly]',
  '[monotone]',
  '[dramatic]',
  '[matter of fact]',
];

Reaction Tags#

typescript
const reactionTags = [
  '[sighs]',
  '[laughs]',
  '[nervous laugh]',
  '[gasps]',
  '[gulps]',
  '[clears throat]',
  '[coughs]',
  '[sniffles]',
  '[yawns]',
  '[groans]',
  '[exhales]',
  '[inhales]',
];

Using Audio Tags#

typescript
const text =
  '[calm] Welcome to this meditation. [pauses] ' +
  'Take a deep breath. [exhales] ' +
  'Let your body relax. [slowly] ' +
  'Feel the peace within. [serene]';

const response = await provider.textToSpeech({
  text,
  voiceId: 'voice_id',
  modelId: 'eleven_multilingual_v2', // v3 model supports tags
});

Parse Audio Tags#

typescript
import { parseAudioTags } from './providers/elevenlabs-audio-tags';

const result = parseAudioTags('[calm] Hello [pauses] there.');

console.log(result.cleanText); // "Hello there."
console.log(result.tags);
// [
//   { tag: '[calm]', innerText: 'calm', category: 'emotion', isValid: true },
//   { tag: '[pauses]', innerText: 'pauses', category: 'cognitive', isValid: true }
// ]

Validate Tags#

typescript
import {
  validateTag,
  validateTaggedText,
} from './providers/elevenlabs-audio-tags';

const validation = validateTag('whispers');
// { tag: 'whispers', isValid: true, category: 'delivery' }

const result = validateTaggedText('[calm] Hello [unknown tag] world');
// { isValid: true, warnings: ['Tag "[unknown tag]": Custom tag (not in known list)'] }

Audio Tag Templates#

Pre-built templates for different content types:

typescript
import {
  MEDITATION_TEMPLATE,
  SLEEP_TEMPLATE,
  ENERGIZING_TEMPLATE,
  STORYTELLING_TEMPLATE,
  EDUCATIONAL_TEMPLATE,
} from './providers/elevenlabs-audio-tags';

Meditation Template:

typescript
{
  openingTags: ['[calm]', '[serene]', '[gently]'],
  transitionTags: ['[pauses]', '[softly]', '[deep breath]'],
  closingTags: ['[tenderly]', '[peacefully]', '[slowly]'],
  paragraphTags: ['[calmly]', '[serenely]'],
  emphasisTags: ['[gently]', '[lovingly]'],
  pauseTags: ['[pauses]', '[long pause]', '[exhales]']
}

Sleep Template:

typescript
{
  openingTags: ['[slowly]', '[softly]', '[whispers]'],
  transitionTags: ['[long pause]', '[exhales]', '[yawns]'],
  closingTags: ['[whispers]', '[trails off]', '[slowly]'],
  paragraphTags: ['[softly]', '[murmurs]'],
  emphasisTags: ['[tenderly]', '[whispers]'],
  pauseTags: ['[long pause]', '[pauses]', '[exhales deeply]']
}

Inject Tags Automatically#

typescript
import {
  injectMeditationTags,
  injectSleepTags,
  injectEnergizingTags,
} from './providers/elevenlabs-audio-tags';

const meditation = injectMeditationTags(
  'Welcome to this practice. Close your eyes.',
  { pauseFrequency: 2 }
);
// "[calm] Welcome to this practice. [pauses] Close your eyes. [tenderly]"

const sleep = injectSleepTags('Let your body relax. Feel yourself sinking.');
// "[slowly] Let your body relax. [long pause] Feel yourself sinking. [whispers]"

Custom Tag Injection#

typescript
import { injectAudioTags } from './providers/elevenlabs-audio-tags';

const result = injectAudioTags(text, {
  template: MEDITATION_TEMPLATE,
  addOpeningTag: true,
  addClosingTag: true,
  addTransitions: true,
  pauseAfterSentences: true,
  pauseFrequency: 3,
  addEmphasis: true,
  emphasisKeywords: ['peace', 'calm', 'relax'],
  customReplacements: new Map([['breath', '[inhales] breath']]),
});

Utility Functions#

typescript
import {
  extractTags,
  removeAudioTags,
  countAudioTags,
  hasAudioTags,
  formatTag,
  combineTags,
  getTagsByCategory,
  getSuggestedTags,
} from './providers/elevenlabs-audio-tags';

// Extract tags
const tags = extractTags('[calm] Hello [pauses]');
// ['[calm]', '[pauses]']

// Remove tags
const clean = removeAudioTags('[calm] Hello [pauses]');
// 'Hello'

// Count tags
const count = countAudioTags('[calm] Hello [pauses]');
// 2

// Check for tags
const hasTags = hasAudioTags('[calm] Hello');
// true

// Format tag
const formatted = formatTag('whispers');
// '[whispers]'

// Combine tags
const combined = combineTags('calm', 'slowly');
// '[calm][slowly]'

// Get tags by category
const emotions = getTagsByCategory('emotion');

// Get suggested tags for content type
const suggestions = getSuggestedTags('meditation');

Conversational AI (WebRTC/WebSocket)#

Real-time conversational AI with bidirectional audio for interactive voice agents.

Connection Types#

Type Description Use Case
websocket WebSocket connection Server-side, full control
webrtc WebRTC peer connection Client-side, browser

Get Conversation Token (WebRTC)#

typescript
const tokenResponse = await provider.getConversationToken(
  'agent_abc123',
  'User Name' // Optional participant name
);

console.log('Token:', tokenResponse.token);
// Use token to establish WebRTC connection

Get Signed WebSocket URL#

typescript
const urlResponse = await provider.getConversationSignedUrl('agent_abc123');

const ws = new WebSocket(urlResponse.signedUrl);

Get Public Agent WebSocket URL#

For public agents that don't require authentication:

typescript
const wsUrl = provider.getPublicAgentWebSocketUrl('agent_abc123');
// 'wss://api.elevenlabs.io/v1/convai/conversation?agent_id=agent_abc123'

Create Conversation Session#

typescript
const session = provider.createConversationSession({
  agentId: 'agent_abc123',
  connectionType: 'websocket',
  participantName: 'User',
  initialContext: 'The user is interested in meditation.',
  audioFormat: 'pcm',
  sampleRate: 16000,
});

console.log('Session ID:', session.conversationId);
console.log('Status:', session.status); // 'connecting'
console.log('Mode:', session.mode); // 'listening'

Conversation Configuration#

typescript
interface ConversationConfig {
  agentId: string; // Agent ID (required)
  connectionType?: 'websocket' | 'webrtc'; // Connection type
  participantName?: string; // User's name
  customMetadata?: Record<string, unknown>; // Custom data
  initialContext?: string; // Initial context
  audioFormat?: 'pcm' | 'mp3' | 'opus'; // Audio format
  sampleRate?: number; // Sample rate (Hz)
}

Conversation Session#

typescript
interface ConversationSession {
  conversationId: string;
  agentId: string;
  connectionType: 'websocket' | 'webrtc';
  status: ConversationStatus;
  mode: ConversationMode;
  startedAt: Date;
  participantName?: string;
}

type ConversationStatus = 'connecting' | 'connected' | 'disconnected' | 'error';
type ConversationMode = 'speaking' | 'listening';

Server-to-Client Events#

Event Type Description
user_transcript User speech transcription
agent_response Agent's text response
agent_response_correction Corrected agent response
audio Agent audio chunk
interruption User interrupted agent
ping Keep-alive ping
conversation_initiation_metadata Session metadata

Parse Server Events#

typescript
const event = provider.parseServerEvent(rawMessage);

if (event) {
  switch (event.type) {
    case 'user_transcript':
      console.log('User said:', event.userTranscript);
      console.log('Final:', event.isFinal);
      break;
    case 'agent_response':
      console.log('Agent:', event.agentResponse);
      break;
    case 'audio':
      const audioBuffer = Buffer.from(event.audioBase64, 'base64');
      playAudio(audioBuffer);
      break;
    case 'ping':
      ws.send(JSON.stringify(provider.createPongEvent(event.eventId)));
      break;
    case 'interruption':
      console.log('Interrupted:', event.reason);
      break;
    case 'conversation_initiation_metadata':
      console.log('Conversation ID:', event.conversationId);
      break;
  }
}

Server Event Types#

typescript
interface UserTranscriptEvent {
  type: 'user_transcript';
  userTranscript: string;
  isFinal: boolean;
}

interface AgentResponseEvent {
  type: 'agent_response';
  agentResponse: string;
}

interface ConversationAudioEvent {
  type: 'audio';
  audioBase64: string;
  eventId: number;
}

interface InterruptionEvent {
  type: 'interruption';
  reason: string;
}

interface PingEvent {
  type: 'ping';
  eventId: number;
  pingMs?: number;
}

interface ConversationInitiationMetadata {
  type: 'conversation_initiation_metadata';
  conversationId: string;
  agentConfig?: Record<string, unknown>;
}

Client-to-Server Events#

Send Audio Chunk#

typescript
// Audio: PCM 16-bit mono at 16kHz, base64 encoded
const audioEvent = provider.createUserAudioChunkEvent(audioBase64);
ws.send(JSON.stringify(audioEvent));

Send Text Message#

typescript
const textEvent = provider.createUserTextMessageEvent('Hello, how are you?');
ws.send(JSON.stringify(textEvent));

Send Contextual Update#

Non-interrupting information for the agent:

typescript
const contextEvent = provider.createContextualUpdateEvent(
  'User just started a breathing exercise'
);
ws.send(JSON.stringify(contextEvent));

Send User Activity#

Prevent agent interruption during user actions:

typescript
const activityEvent = provider.createUserActivityEvent();
ws.send(JSON.stringify(activityEvent));

Send Pong Response#

Reply to ping events:

typescript
const pongEvent = provider.createPongEvent(pingEvent.eventId);
ws.send(JSON.stringify(pongEvent));

Client Event Types#

typescript
interface ContextualUpdateEvent {
  type: 'contextual_update';
  text: string;
}

interface UserAudioChunkEvent {
  userAudioChunk: string; // base64 PCM
}

interface UserTextMessageEvent {
  type: 'user_message';
  text: string;
}

interface UserActivityEvent {
  type: 'user_activity';
}

interface PongEvent {
  type: 'pong';
  eventId: number;
}

Conversation Messages#

Create and track conversation messages:

typescript
const message = provider.createConversationMessage(
  'user', // 'user' or 'agent'
  'Hello there!', // text
  true, // isFinal
  audioBase64 // optional audio
);

// {
//   id: 'msg_1704067200_abc123',
//   role: 'user',
//   text: 'Hello there!',
//   timestamp: Date,
//   isFinal: true,
//   audioBase64: '...'
// }

Conversation History#

typescript
const history = provider.createConversationHistory(
  'conv_123',
  messages,
  session.startedAt,
  new Date() // endedAt
);

// {
//   conversationId: 'conv_123',
//   messages: [...],
//   durationSec: 120,
//   startedAt: Date,
//   endedAt: Date
// }

Complete WebSocket Example#

typescript
async function startConversation(agentId: string) {
  // 1. Get signed URL
  const { signedUrl } = await provider.getConversationSignedUrl(agentId);

  // 2. Create session
  const session = provider.createConversationSession({ agentId });
  const messages: ConversationMessage[] = [];

  // 3. Connect WebSocket
  const ws = new WebSocket(signedUrl);

  ws.onopen = () => {
    console.log('Connected');
    session.status = 'connected';
  };

  ws.onmessage = (event) => {
    const parsed = provider.parseServerEvent(event.data);
    if (!parsed) return;

    switch (parsed.type) {
      case 'user_transcript':
        if (parsed.isFinal) {
          messages.push(
            provider.createConversationMessage(
              'user',
              parsed.userTranscript,
              true
            )
          );
        }
        break;

      case 'agent_response':
        messages.push(
          provider.createConversationMessage(
            'agent',
            parsed.agentResponse,
            true
          )
        );
        break;

      case 'audio':
        // Play audio through speakers
        playAudio(Buffer.from(parsed.audioBase64, 'base64'));
        break;

      case 'ping':
        ws.send(JSON.stringify(provider.createPongEvent(parsed.eventId)));
        break;
    }
  };

  ws.onclose = () => {
    session.status = 'disconnected';
    const history = provider.createConversationHistory(
      session.conversationId,
      messages,
      session.startedAt,
      new Date()
    );
    console.log('Conversation duration:', history.durationSec, 'seconds');
  };

  // 4. Send user audio (from microphone)
  startMicrophoneCapture((audioChunk) => {
    const base64 = audioChunk.toString('base64');
    ws.send(JSON.stringify(provider.createUserAudioChunkEvent(base64)));
  });

  // 5. Send context updates as needed
  ws.send(
    JSON.stringify(
      provider.createContextualUpdateEvent(
        'User is in a quiet room ready for meditation guidance'
      )
    )
  );
}

Sound Effects#

Generate AI sound effects from text descriptions for meditation and ambient content.

Generate Sound Effect#

typescript
const response = await provider.generateSoundEffect({
  text: 'A Tibetan singing bowl being gently struck',
  durationSeconds: 8,
  promptInfluence: 0.5,
  loop: false,
  outputFormat: 'mp3_44100_128',
});

fs.writeFileSync('singing-bowl.mp3', response.audio);

Sound Effects Request#

typescript
interface SoundEffectsRequest {
  text: string; // Description (required)
  modelId?: SoundEffectsModelId; // Model to use
  durationSeconds?: number; // 0.5-30 seconds
  promptInfluence?: number; // 0-1 (default 0.3)
  loop?: boolean; // Seamless loop (v2 only)
  outputFormat?: OutputFormat; // Audio format
}

type SoundEffectsModelId =
  | 'eleven_text_to_sound_v1'
  | 'eleven_text_to_sound_v2';

Sound Effects Response#

typescript
interface SoundEffectsResponse {
  audio: Buffer; // Generated audio
  contentType: string; // MIME type
  requestId?: string; // For tracking
  durationSeconds?: number;
  isLooped?: boolean;
}

Generate from Preset#

typescript
const response = await provider.generateSoundEffectFromPreset(
  'Tibetan Singing Bowl',
  {
    durationSeconds: 10, // Override preset duration
    loop: true, // Override preset loop setting
  }
);

Batch Generation#

typescript
// Multiple custom requests
const responses = await provider.generateSoundEffectBatch([
  { text: 'Gentle rain', durationSeconds: 20, loop: true },
  { text: 'Temple bell', durationSeconds: 6 },
  { text: 'Wind chimes', durationSeconds: 10 },
]);

// Multiple presets
const presetResults = await provider.generateSoundEffectBatchFromPresets(
  ['Tibetan Singing Bowl', 'Ocean Waves', 'Forest Birds'],
  { outputFormat: 'mp3_44100_128' }
);

for (const result of presetResults) {
  if (result.response) {
    fs.writeFileSync(`${result.preset}.mp3`, result.response.audio);
  } else {
    console.log(`Error generating ${result.preset}:`, result.error);
  }
}

Sound Effects Presets#

17 built-in presets for meditation and ambient content.

Preset Categories#

Category Presets
meditation Tibetan Singing Bowl, Temple Bell, Meditation Chime, Om Drone
nature Gentle Rain, Ocean Waves, Forest Birds, Flowing Stream, Wind in Trees, Campfire Crackling
ambient Ethereal Pad, Crystal Resonance, Deep Space
transition Breath Transition, Shimmer, Gong Wash
notification Mindful Bell, Soft Chime

Available Presets#

Preset Duration Loop Category
Tibetan Singing Bowl 8s No meditation
Temple Bell 6s No meditation
Meditation Chime 4s No meditation
Om Drone 20s Yes meditation
Gentle Rain 20s Yes nature
Ocean Waves 20s Yes nature
Forest Birds 20s Yes nature
Flowing Stream 20s Yes nature
Wind in Trees 20s Yes nature
Campfire Crackling 20s Yes nature
Ethereal Pad 20s Yes ambient
Crystal Resonance 15s Yes ambient
Deep Space 20s Yes ambient
Breath Transition 2s No transition
Shimmer 2s No transition
Gong Wash 8s No transition
Mindful Bell 3s No notification
Soft Chime 2s No notification

Preset Structure#

typescript
interface SoundEffectsPreset {
  name: string;
  description: string;
  category:
    | 'meditation'
    | 'nature'
    | 'ambient'
    | 'transition'
    | 'notification'
    | 'musical'
    | 'custom';
  prompt: string;
  recommendedDuration: number;
  shouldLoop: boolean;
  promptInfluence: number;
}

Preset Utilities#

typescript
import {
  SOUND_EFFECTS_PRESETS,
  getSoundEffectsPreset,
  getSoundEffectsPresetsByCategory,
  createSoundEffectsPreset,
} from './providers/elevenlabs-provider';

// Get preset by name
const preset = getSoundEffectsPreset('Tibetan Singing Bowl');

// Get presets by category
const naturePresets = getSoundEffectsPresetsByCategory('nature');

// Create custom preset
const customPreset = createSoundEffectsPreset(
  'Zen Garden',
  'Water feature trickling in a peaceful zen garden with soft wind chimes',
  {
    category: 'ambient',
    description: 'Peaceful zen garden ambiance',
    recommendedDuration: 20,
    shouldLoop: true,
    promptInfluence: 0.4,
  }
);

Music Generation#

Generate AI music from text descriptions for meditation and wellness content.

Generate Music#

typescript
const response = await provider.generateMusic({
  prompt: 'Create slow, ambient meditation music with soft synthesizer pads',
  durationMs: 180000, // 3 minutes
  includeVocals: false,
});

fs.writeFileSync('meditation-music.mp3', response.audio);

Music Generation Request#

typescript
interface MusicGenerationRequest {
  prompt: string; // Description (required)
  durationMs?: number; // 10000-300000ms (10s-5min)
  compositionPlan?: MusicCompositionPlan; // Structured plan
  includeVocals?: boolean; // Include vocals
  vocalsLanguage?: 'en' | 'es' | 'de' | 'ja' | 'fr' | 'pt' | 'it' | 'zh';
}

Music Generation Response#

typescript
interface MusicGenerationResponse {
  audio: Buffer; // MP3 audio
  contentType: string; // 'audio/mpeg'
  requestId?: string; // For tracking
  durationMs?: number;
  filename?: string;
  compositionPlan?: MusicCompositionPlan; // If detailed
}

Generate with Composition Plan#

typescript
// First generate a composition plan
const plan = await provider.generateCompositionPlan({
  prompt: 'Peaceful meditation music with a gentle intro and calming outro',
  durationMs: 120000,
});

// Then generate music with the plan
const response = await provider.generateMusic({
  prompt: 'Meditation music',
  durationMs: 120000,
  compositionPlan: plan,
});

Composition Plan Structure#

typescript
interface MusicCompositionPlan {
  positiveGlobalStyles: string[]; // Global positive attributes
  negativeGlobalStyles: string[]; // Styles to avoid
  sections: MusicSection[]; // Track structure
}

interface MusicSection {
  sectionName: string; // 'intro', 'verse', 'outro', etc.
  positiveLocalStyles: string[]; // Section-specific attributes
  negativeLocalStyles: string[]; // Section-specific avoids
  durationMs: number; // Section duration
  lines?: string[]; // Lyrics (if vocals)
}

Detailed Generation#

Get both audio and composition plan:

typescript
const response = await provider.generateMusicDetailed({
  prompt: 'Ambient yoga music with soft sitar and tabla',
  durationMs: 180000,
});

console.log('Plan:', response.compositionPlan);
console.log('Filename:', response.filename);

Generate from Preset#

typescript
const response = await provider.generateMusicFromPreset('Deep Meditation', {
  durationMs: 300000, // Override to 5 minutes
});

Batch Generation#

typescript
// Multiple custom tracks
const tracks = await provider.generateMusicBatch([
  { prompt: 'Morning meditation', durationMs: 180000 },
  { prompt: 'Evening relaxation', durationMs: 240000 },
]);

// Multiple presets
const presetTracks = await provider.generateMusicBatchFromPresets(
  ['Deep Meditation', 'Yoga Flow', 'Deep Sleep'],
  { durationMs: 180000 }
);

for (const result of presetTracks) {
  if (result.response) {
    fs.writeFileSync(`${result.preset}.mp3`, result.response.audio);
  }
}

Music Generation Presets#

14 built-in presets for meditation, yoga, sleep, and spiritual content.

Preset Categories#

Category Presets
meditation Deep Meditation, Guided Meditation Background, Zen Garden
yoga Yoga Flow, Restorative Yoga
sleep Deep Sleep, Lullaby Dreams
focus Focus Flow
spiritual Sacred Om, Temple Atmosphere
nature Forest Meditation, Ocean Waves Music
relaxation Spa Relaxation, Evening Unwind

Available Presets#

Preset Duration Vocals BPM Category
Deep Meditation 5min No 50-70 meditation
Guided Meditation Background 5min No 50-60 meditation
Zen Garden 4min No 50-65 meditation
Yoga Flow 5min No 75-95 yoga
Restorative Yoga 5min No 45-60 yoga
Deep Sleep 5min No 40-55 sleep
Lullaby Dreams 4min No 50-65 sleep
Focus Flow 5min No 65-85 focus
Sacred Om 5min Yes 55-70 spiritual
Temple Atmosphere 4min No 50-65 spiritual
Forest Meditation 4min No 50-65 nature
Ocean Waves Music 4min No 50-60 nature
Spa Relaxation 4min No 55-70 relaxation
Evening Unwind 4min No 50-65 relaxation

Preset Structure#

typescript
interface MusicGenerationPreset {
  name: string;
  description: string;
  category:
    | 'meditation'
    | 'yoga'
    | 'sleep'
    | 'focus'
    | 'relaxation'
    | 'spiritual'
    | 'nature'
    | 'custom';
  prompt: string;
  recommendedDurationMs: number;
  includeVocals: boolean;
  bpmRange?: { min: number; max: number };
  musicalElements?: string[];
}

Preset Utilities#

typescript
import {
  MUSIC_GENERATION_PRESETS,
  getMusicGenerationPreset,
  getMusicGenerationPresetsByCategory,
  createMusicGenerationPreset,
} from './providers/elevenlabs-provider';

// Get preset by name
const preset = getMusicGenerationPreset('Deep Meditation');

// Get presets by category
const sleepPresets = getMusicGenerationPresetsByCategory('sleep');

// Create custom preset
const customPreset = createMusicGenerationPreset(
  'Morning Awakening',
  'Gentle, uplifting music to start the day with soft strings and light percussion',
  {
    category: 'meditation',
    description: 'Energizing morning music',
    recommendedDurationMs: 180000,
    includeVocals: false,
    bpmRange: { min: 65, max: 80 },
    musicalElements: ['strings', 'light percussion', 'soft piano'],
  }
);

Dubbing#

Translate and dub audio/video content into multiple languages.

Create Dubbing Job#

typescript
// From file
const response = await provider.createDubbing({
  file: audioBuffer,
  filename: 'meditation.mp3',
  targetLang: 'es',
  sourceLang: 'en',
  name: 'Meditation Guide - Spanish',
});

// From URL
const response = await provider.createDubbing({
  sourceUrl: 'https://example.com/video.mp4',
  targetLang: 'fr',
  name: 'Tutorial - French',
});

console.log('Dubbing ID:', response.dubbingId);
console.log('Expected duration:', response.expectedDurationSec, 'seconds');

Dubbing Request#

typescript
interface DubbingRequest {
  targetLang: DubbingLanguage | string; // Target language (required)
  sourceLang?: DubbingLanguage | string; // Source (default: auto-detect)
  file?: Buffer; // Audio/video file
  filename?: string; // Filename for file
  sourceUrl?: string; // Alternative to file
  name?: string; // Project name
  targetAccent?: string; // Accent preference
  numSpeakers?: number; // 0 = auto-detect (max 9)
  mode?: DubbingMode; // 'automatic' | 'manual'
  watermark?: boolean; // Apply watermark
  highestResolution?: boolean; // Use highest resolution
  dropBackgroundAudio?: boolean; // Remove background
  useProfanityFilter?: boolean; // Censor profanities
  disableVoiceCloning?: boolean; // Use voice library
  dubbingStudio?: boolean; // Enable studio edits
  startTime?: number; // Clip start (ms)
  endTime?: number; // Clip end (ms)
}

Dubbing Create Response#

typescript
interface DubbingCreateResponse {
  dubbingId: string;
  expectedDurationSec: number;
}

Get Dubbing Status#

typescript
const metadata = await provider.getDubbingMetadata(dubbingId);

console.log('Status:', metadata.status); // 'dubbing' | 'dubbed' | 'failed'
console.log('Target Languages:', metadata.targetLanguages);
console.log('Editable:', metadata.editable);

Dubbing Metadata#

typescript
interface DubbingMetadata {
  dubbingId: string;
  name: string;
  status: DubbingStatus;
  targetLanguages: string[];
  editable: boolean;
  createdAt: Date;
  mediaMetadata?: {
    contentType: string;
    duration: number;
  };
  error?: string;
}

type DubbingStatus = 'dubbing' | 'dubbed' | 'failed' | 'cloning';

Wait for Completion#

typescript
const metadata = await provider.waitForDubbing(dubbingId, {
  pollIntervalMs: 5000, // Check every 5 seconds
  timeoutMs: 600000, // 10 minute timeout
});

if (metadata.status === 'dubbed') {
  console.log('Dubbing complete!');
} else if (metadata.status === 'failed') {
  console.log('Error:', metadata.error);
}

Get Dubbed Audio#

typescript
const audio = await provider.getDubbedAudio(dubbingId, 'es');

fs.writeFileSync('meditation-spanish.mp3', audio.audio);
console.log('Content type:', audio.contentType);
console.log('Language:', audio.languageCode);

Dubbed Audio Response#

typescript
interface DubbedAudioResponse {
  audio: Buffer;
  contentType: string;
  languageCode: string;
}

Get Transcript#

typescript
const transcript = await provider.getDubbingTranscript(dubbingId, 'es');

console.log('Full text:', transcript.text);

for (const segment of transcript.segments) {
  console.log(`[${segment.startMs}-${segment.endMs}ms] ${segment.text}`);
  if (segment.words) {
    for (const word of segment.words) {
      console.log(`  ${word.word} (${word.confidence})`);
    }
  }
}

Transcript Response#

typescript
interface DubbingTranscriptResponse {
  languageCode: string;
  text: string;
  segments: TranscriptSegment[];
}

interface TranscriptSegment {
  text: string;
  speakerId?: string;
  startMs: number;
  endMs: number;
  words?: TranscriptWord[];
}

interface TranscriptWord {
  word: string;
  startMs: number;
  endMs: number;
  confidence?: number;
}

Delete Dubbing Project#

typescript
await provider.deleteDubbing(dubbingId);

Supported Dubbing Languages#

32 languages supported for dubbing.

Code Language Code Language
en English ko Korean
es Spanish hu Hungarian
fr French hi Hindi
de German sv Swedish
it Italian da Danish
pt Portuguese fi Finnish
pl Polish no Norwegian
tr Turkish el Greek
ru Russian he Hebrew
nl Dutch ro Romanian
cs Czech uk Ukrainian
ar Arabic id Indonesian
zh Chinese ms Malay
ja Japanese vi Vietnamese
th Thai bg Bulgarian
sk Slovak hr Croatian

Language Utilities#

typescript
import {
  DUBBING_LANGUAGES,
  getDubbingLanguageName,
  isDubbingLanguageSupported,
} from './providers/elevenlabs-provider';

// Get language name
const name = getDubbingLanguageName('ja'); // 'Japanese'

// Check if supported
if (isDubbingLanguageSupported('es')) {
  console.log('Spanish is supported');
}

// List all languages
for (const [code, name] of Object.entries(DUBBING_LANGUAGES)) {
  console.log(`${code}: ${name}`);
}

Error Handling#

ElevenLabsError#

typescript
interface ElevenLabsError extends Error {
  code: string;
  status: number;
  detail?: string;
}

Error Handling Example#

typescript
try {
  const response = await provider.textToSpeech({
    text: 'Hello',
    voiceId: 'invalid_id',
  });
} catch (error) {
  if (error.status === 401) {
    console.log('Invalid API key');
  } else if (error.status === 429) {
    console.log('Rate limited, retry after cooldown');
  } else if (error.status === 400) {
    console.log('Bad request:', error.detail);
  }
}