Eve library

Audio Streaming Guide

Guide

Revision 1

On this page

Comprehensive guide to implementing high-quality audio streaming and text-to-speech synthesis with the Lilith API.

Table of Contents#

Overview#

The Lilith Audio API provides professional-grade text-to-speech synthesis optimized for wisdom and educational content. Features include multiple high-quality voices, SSML support, real-time streaming, and cross-platform compatibility.

Key Features#

  • 🎡 High-Quality Voices: Neural TTS with natural prosody
  • ⚑ Real-time Streaming: Low-latency audio generation
  • πŸ”§ SSML Support: Advanced speech control and customization
  • πŸ“± Multi-format: Opus, MP3, WAV, and streaming formats
  • 🌐 Cross-platform: Web, mobile, and desktop support
  • 🎯 Content-Optimized: Specially trained for wisdom content
  • πŸ“Š Analytics: Detailed audio metrics and monitoring

Audio Pipeline#

FlowchartFlowchart: 10 steps and 10 connections.StreamingFileText/SSML InputContent AnalysisVoice SelectionNeural TTS EngineOutput Mode?Real-time ChunksComplete AudioHLS StreamingCDN DistributionClient Playback
Flowchart: Flowchart. Flowchart: 10 steps and 10 connections.

Drawn from its Mermaid source.

As text
Parts:
- Text/SSML Input
- Content Analysis
- Voice Selection
- Neural TTS Engine
- Output Mode? (decision)
- Real-time Chunks
- Complete Audio
- HLS Streaming
- CDN Distribution
- Client Playback

Connections:
- Text/SSML Input to Content Analysis
- Content Analysis to Voice Selection
- Voice Selection to Neural TTS Engine
- Neural TTS Engine to Output Mode?
- Output Mode? to Real-time Chunks: Streaming
- Output Mode? to Complete Audio: File
- Real-time Chunks to HLS Streaming
- Complete Audio to CDN Distribution
- HLS Streaming to Client Playback
- CDN Distribution to Client Playback
Show source
graph TD
    A[Text/SSML Input] --> B[Content Analysis]
    B --> C[Voice Selection]
    C --> D[Neural TTS Engine]
    D --> E{Output Mode?}
    E -->|Streaming| F[Real-time Chunks]
    E -->|File| G[Complete Audio]
    F --> H[HLS Streaming]
    G --> I[CDN Distribution]
    H --> J[Client Playback]
    I --> J

Quick Start#

Basic Text-to-Speech#

bash
curl -X POST "https://api.lilith.app/v1/tts/synthesize" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "voice_id": "calm_female_01",
    "ssml": "<speak>Welcome to your daily mindfulness practice.</speak>",
    "format": "opus",
    "stream": false
  }'

Simple Integration Example#

javascript
class LilithAudio {
  constructor(apiToken) {
    this.apiToken = apiToken;
    this.baseURL = 'https://api.lilith.app';
  }

  async synthesizeSpeech(text, voiceId = 'calm_female_01', options = {}) {
    const response = await fetch(`${this.baseURL}/v1/tts/synthesize`, {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${this.apiToken}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        voice_id: voiceId,
        ssml: `<speak>${text}</speak>`,
        format: options.format || 'opus',
        stream: options.stream || false,
        speed: options.speed || 1.0,
        pitch: options.pitch || 0.0,
      }),
    });

    if (!response.ok) {
      throw new Error(`TTS API error: ${response.status}`);
    }

    if (options.stream) {
      return response.body; // ReadableStream
    } else {
      const result = await response.json();
      return result.audio_url;
    }
  }

  async playAudio(audioUrl) {
    const audio = new Audio(audioUrl);
    return new Promise((resolve, reject) => {
      audio.onended = resolve;
      audio.onerror = reject;
      audio.play();
    });
  }
}

// Usage
const audio = new LilithAudio(apiToken);

try {
  const audioUrl = await audio.synthesizeSpeech(
    'Breathe in deeply, hold for three counts, then slowly exhale.',
    'calm_female_01'
  );

  await audio.playAudio(audioUrl);
  console.log('Audio playback completed');
} catch (error) {
  console.error('Audio error:', error);
}

Audio Synthesis#

Voice Selection#

Choose from our curated collection of voices optimized for different content types:

javascript
const VoiceProfiles = {
  // Meditation and mindfulness content
  CALM_FEMALE: 'calm_female_01',
  GENTLE_FEMALE: 'gentle_female_05',

  // Educational and scholarly content
  SCHOLARLY_MALE: 'scholarly_male_02',
  WARM_FEMALE: 'warm_female_03',

  // Philosophical and contemplative content
  DEEP_MALE: 'deep_male_04',

  // Conversational and friendly content
  FRIENDLY_MALE: 'friendly_male_06',
  BRIGHT_FEMALE: 'bright_female_07',
};

class VoiceSelector {
  static selectVoice(contentType, userPreference = null) {
    if (userPreference && this.isValidVoice(userPreference)) {
      return userPreference;
    }

    const voiceMap = {
      meditation: VoiceProfiles.CALM_FEMALE,
      philosophy: VoiceProfiles.DEEP_MALE,
      education: VoiceProfiles.SCHOLARLY_MALE,
      conversation: VoiceProfiles.WARM_FEMALE,
      storytelling: VoiceProfiles.GENTLE_FEMALE,
    };

    return voiceMap[contentType] || VoiceProfiles.CALM_FEMALE;
  }

  static isValidVoice(voiceId) {
    return Object.values(VoiceProfiles).includes(voiceId);
  }

  static getVoiceInfo(voiceId) {
    const voiceDetails = {
      calm_female_01: {
        name: 'Serena',
        gender: 'female',
        age: 'adult',
        accent: 'neutral',
        characteristics: ['calm', 'soothing', 'meditative'],
        bestFor: ['meditation', 'mindfulness', 'relaxation'],
      },
      scholarly_male_02: {
        name: 'Professor Marcus',
        gender: 'male',
        age: 'mature',
        accent: 'neutral',
        characteristics: ['authoritative', 'clear', 'educational'],
        bestFor: ['lectures', 'philosophy', 'academic content'],
      },
      warm_female_03: {
        name: 'Elena',
        gender: 'female',
        age: 'adult',
        accent: 'neutral',
        characteristics: ['warm', 'conversational', 'friendly'],
        bestFor: ['dialogue', 'conversation', 'storytelling'],
      },
      // ... other voice profiles
    };

    return voiceDetails[voiceId] || null;
  }
}

Advanced Speech Configuration#

javascript
class SpeechConfiguration {
  constructor() {
    this.defaultSettings = {
      speed: 1.0, // 0.5 - 2.0
      pitch: 0.0, // -20.0 to +20.0 semitones
      volume: 1.0, // 0.0 - 1.0
      emphasis: 'none', // none, moderate, strong, reduced
      pauseDuration: 'medium', // short, medium, long
    };
  }

  createConfiguration(contentType, userPreferences = {}) {
    const configurations = {
      meditation: {
        speed: 0.8,
        pitch: -2.0,
        emphasis: 'reduced',
        pauseDuration: 'long',
      },
      lecture: {
        speed: 0.9,
        pitch: 0.0,
        emphasis: 'moderate',
        pauseDuration: 'medium',
      },
      conversation: {
        speed: 1.0,
        pitch: 0.0,
        emphasis: 'moderate',
        pauseDuration: 'short',
      },
      story: {
        speed: 0.95,
        pitch: 1.0,
        emphasis: 'strong',
        pauseDuration: 'medium',
      },
    };

    const baseConfig = configurations[contentType] || this.defaultSettings;
    return { ...baseConfig, ...userPreferences };
  }

  validateSettings(settings) {
    const constraints = {
      speed: { min: 0.5, max: 2.0 },
      pitch: { min: -20.0, max: 20.0 },
      volume: { min: 0.0, max: 1.0 },
    };

    for (const [key, value] of Object.entries(settings)) {
      if (constraints[key]) {
        const { min, max } = constraints[key];
        if (value < min || value > max) {
          throw new Error(`${key} must be between ${min} and ${max}`);
        }
      }
    }

    return true;
  }
}

Streaming Audio#

Real-time Audio Streaming#

javascript
class StreamingAudio {
  constructor(apiToken) {
    this.apiToken = apiToken;
    this.baseURL = 'https://api.lilith.app';
  }

  async streamSpeech(text, voiceId, onChunk, onComplete, onError) {
    try {
      const response = await fetch(`${this.baseURL}/v1/tts/synthesize`, {
        method: 'POST',
        headers: {
          Authorization: `Bearer ${this.apiToken}`,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({
          voice_id: voiceId,
          ssml: `<speak>${text}</speak>`,
          format: 'opus',
          stream: true,
        }),
      });

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      const reader = response.body.getReader();
      const audioContext = new (
        window.AudioContext || window.webkitAudioContext
      )();
      let totalBytes = 0;

      while (true) {
        const { done, value } = await reader.read();

        if (done) {
          onComplete(totalBytes);
          break;
        }

        totalBytes += value.byteLength;
        await this.processAudioChunk(audioContext, value, onChunk);
      }
    } catch (error) {
      onError(error);
    }
  }

  async processAudioChunk(audioContext, chunk, onChunk) {
    try {
      const audioBuffer = await audioContext.decodeAudioData(
        chunk.buffer.slice()
      );
      onChunk(audioBuffer);
    } catch (error) {
      console.warn('Could not decode audio chunk:', error);
      // Continue with raw chunk
      onChunk(chunk);
    }
  }

  // HLS Streaming Support
  async createHLSStream(text, voiceId, options = {}) {
    const response = await fetch(`${this.baseURL}/v1/tts/synthesize`, {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${this.apiToken}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        voice_id: voiceId,
        ssml: `<speak>${text}</speak>`,
        format: 'hls',
        stream: true,
        ...options,
      }),
    });

    const result = await response.json();
    return result.stream_url; // HLS manifest URL
  }
}

// Usage example
const streaming = new StreamingAudio(apiToken);

let audioQueue = [];
let currentSource = null;

streaming.streamSpeech(
  'This is a long text that will be streamed as audio chunks for real-time playback.',
  'calm_female_01',

  // On chunk received
  (audioBuffer) => {
    audioQueue.push(audioBuffer);
    if (!currentSource) {
      playNextChunk();
    }
  },

  // On completion
  (totalBytes) => {
    console.log(`Streaming completed: ${totalBytes} bytes`);
  },

  // On error
  (error) => {
    console.error('Streaming error:', error);
  }
);

function playNextChunk() {
  if (audioQueue.length === 0) {
    currentSource = null;
    return;
  }

  const audioContext = new (window.AudioContext || window.webkitAudioContext)();
  const buffer = audioQueue.shift();

  currentSource = audioContext.createBufferSource();
  currentSource.buffer = buffer;
  currentSource.connect(audioContext.destination);

  currentSource.onended = () => {
    playNextChunk(); // Play next chunk when current ends
  };

  currentSource.start();
}

Progressive Audio Loading#

javascript
class ProgressiveAudioPlayer {
  constructor() {
    this.audio = new Audio();
    this.chunks = [];
    this.mediaSource = null;
    this.sourceBuffer = null;
    this.isPlaying = false;
  }

  async initializeMediaSource() {
    if (!window.MediaSource) {
      throw new Error('MediaSource API not supported');
    }

    this.mediaSource = new MediaSource();
    this.audio.src = URL.createObjectURL(this.mediaSource);

    return new Promise((resolve, reject) => {
      this.mediaSource.addEventListener('sourceopen', () => {
        try {
          this.sourceBuffer = this.mediaSource.addSourceBuffer(
            'audio/webm; codecs="opus"'
          );
          this.sourceBuffer.addEventListener(
            'updateend',
            this.onUpdateEnd.bind(this)
          );
          resolve();
        } catch (error) {
          reject(error);
        }
      });

      this.mediaSource.addEventListener('error', reject);
    });
  }

  async appendAudioChunk(chunk) {
    if (!this.sourceBuffer) {
      throw new Error('MediaSource not initialized');
    }

    if (this.sourceBuffer.updating) {
      // Queue chunk if source buffer is busy
      this.chunks.push(chunk);
      return;
    }

    this.sourceBuffer.appendBuffer(chunk);
  }

  onUpdateEnd() {
    // Process queued chunks
    if (this.chunks.length > 0) {
      const nextChunk = this.chunks.shift();
      this.sourceBuffer.appendBuffer(nextChunk);
    }

    // Start playback if not already playing
    if (!this.isPlaying && this.audio.readyState >= 2) {
      this.play();
    }
  }

  async play() {
    try {
      await this.audio.play();
      this.isPlaying = true;
    } catch (error) {
      console.error('Playback failed:', error);
    }
  }

  pause() {
    this.audio.pause();
    this.isPlaying = false;
  }

  finalize() {
    if (this.mediaSource && this.mediaSource.readyState === 'open') {
      this.mediaSource.endOfStream();
    }
  }

  cleanup() {
    this.pause();
    if (this.audio.src) {
      URL.revokeObjectURL(this.audio.src);
    }
    this.chunks = [];
    this.sourceBuffer = null;
    this.mediaSource = null;
  }
}

Voice Options#

Voice Characteristics and Selection#

javascript
class VoiceManager {
  constructor() {
    this.voices = {
      calm_female_01: {
        name: 'Serena',
        description:
          'Gentle, calming voice perfect for meditation and mindfulness',
        gender: 'female',
        age: 'adult',
        accent: 'neutral',
        languages: ['en-US'],
        characteristics: {
          warmth: 0.8,
          authority: 0.4,
          clarity: 0.9,
          calmness: 0.95,
        },
        sampleUrl: 'https://cdn.lilith.app/voices/samples/calm_female_01.mp3',
        bestFor: ['meditation', 'relaxation', 'bedtime_stories'],
        pricing: { tier: 'premium', costPerMinute: 0.02 },
      },

      scholarly_male_02: {
        name: 'Professor Marcus',
        description: 'Authoritative, clear voice ideal for educational content',
        gender: 'male',
        age: 'mature',
        accent: 'neutral',
        languages: ['en-US', 'en-GB'],
        characteristics: {
          warmth: 0.6,
          authority: 0.9,
          clarity: 0.95,
          calmness: 0.7,
        },
        sampleUrl:
          'https://cdn.lilith.app/voices/samples/scholarly_male_02.mp3',
        bestFor: ['lectures', 'philosophy', 'academic', 'presentations'],
        pricing: { tier: 'premium', costPerMinute: 0.025 },
      },

      warm_female_03: {
        name: 'Elena',
        description:
          'Warm, conversational voice great for dialogue and storytelling',
        gender: 'female',
        age: 'adult',
        accent: 'neutral',
        languages: ['en-US'],
        characteristics: {
          warmth: 0.9,
          authority: 0.6,
          clarity: 0.85,
          calmness: 0.8,
        },
        sampleUrl: 'https://cdn.lilith.app/voices/samples/warm_female_03.mp3',
        bestFor: ['conversation', 'storytelling', 'audiobooks'],
        pricing: { tier: 'standard', costPerMinute: 0.015 },
      },

      // Additional voice profiles...
    };
  }

  getVoiceById(voiceId) {
    return this.voices[voiceId] || null;
  }

  findVoicesByCharacteristics(criteria) {
    return Object.entries(this.voices)
      .filter(([id, voice]) => this.matchesCriteria(voice, criteria))
      .map(([id, voice]) => ({ id, ...voice }));
  }

  matchesCriteria(voice, criteria) {
    for (const [key, value] of Object.entries(criteria)) {
      if (key === 'gender' && voice.gender !== value) return false;
      if (key === 'language' && !voice.languages.includes(value)) return false;
      if (key === 'minWarmth' && voice.characteristics.warmth < value)
        return false;
      if (key === 'minAuthority' && voice.characteristics.authority < value)
        return false;
      if (key === 'bestFor' && !voice.bestFor.includes(value)) return false;
    }
    return true;
  }

  recommendVoice(content, context = {}) {
    const contentAnalysis = this.analyzeContent(content);
    const userPrefs = context.userPreferences || {};

    // Score voices based on content and preferences
    const scores = Object.entries(this.voices).map(([id, voice]) => ({
      id,
      voice,
      score: this.calculateVoiceScore(voice, contentAnalysis, userPrefs),
    }));

    scores.sort((a, b) => b.score - a.score);
    return scores[0];
  }

  analyzeContent(content) {
    const contentLower = content.toLowerCase();

    return {
      isEducational: /teach|learn|explain|understand|concept/.test(
        contentLower
      ),
      isMeditative: /meditat|mindful|breath|peace|calm|relax/.test(
        contentLower
      ),
      isPhilosophical: /wisdom|philosophy|meaning|truth|ethic/.test(
        contentLower
      ),
      isConversational: /hello|hi|how are|what do you think/.test(contentLower),
      wordCount: content.split(/\s+/).length,
      complexity: this.estimateComplexity(content),
    };
  }

  calculateVoiceScore(voice, contentAnalysis, userPrefs) {
    let score = 0;

    // Content type scoring
    if (contentAnalysis.isEducational && voice.bestFor.includes('lectures'))
      score += 3;
    if (contentAnalysis.isMeditative && voice.bestFor.includes('meditation'))
      score += 3;
    if (contentAnalysis.isPhilosophical && voice.bestFor.includes('philosophy'))
      score += 3;
    if (
      contentAnalysis.isConversational &&
      voice.bestFor.includes('conversation')
    )
      score += 3;

    // Characteristic scoring
    if (contentAnalysis.isEducational)
      score += voice.characteristics.authority * 2;
    if (contentAnalysis.isMeditative)
      score += voice.characteristics.calmness * 2;

    // User preference scoring
    if (userPrefs.gender && voice.gender === userPrefs.gender) score += 1;
    if (userPrefs.accent && voice.accent === userPrefs.accent) score += 1;

    return score;
  }

  estimateComplexity(content) {
    const avgWordLength =
      content.split(/\s+/).reduce((sum, word) => sum + word.length, 0) /
      content.split(/\s+/).length;

    const sentenceCount = content.split(/[.!?]+/).length;
    const avgSentenceLength = content.split(/\s+/).length / sentenceCount;

    // Simple complexity score based on word length and sentence structure
    return (avgWordLength * 0.3 + avgSentenceLength * 0.1) / 10;
  }
}

// Usage example
const voiceManager = new VoiceManager();

// Find voices for meditation content
const meditationVoices = voiceManager.findVoicesByCharacteristics({
  bestFor: 'meditation',
  minCalmness: 0.8,
});

// Get voice recommendation
const content =
  'Let us begin with a simple breathing exercise. Breathe in slowly through your nose...';
const recommendation = voiceManager.recommendVoice(content, {
  userPreferences: { gender: 'female' },
});

console.log('Recommended voice:', recommendation.voice.name);

SSML Support#

Advanced Speech Markup#

javascript
class SSMLBuilder {
  constructor() {
    this.content = [];
  }

  speak(callback) {
    this.content.push('<speak>');
    callback();
    this.content.push('</speak>');
    return this;
  }

  text(text) {
    this.content.push(this.escapeText(text));
    return this;
  }

  pause(duration) {
    this.content.push(`<break time="${duration}"/>`);
    return this;
  }

  emphasis(text, level = 'moderate') {
    this.content.push(
      `<emphasis level="${level}">${this.escapeText(text)}</emphasis>`
    );
    return this;
  }

  prosody(text, options = {}) {
    const attrs = [];
    if (options.rate) attrs.push(`rate="${options.rate}"`);
    if (options.pitch) attrs.push(`pitch="${options.pitch}"`);
    if (options.volume) attrs.push(`volume="${options.volume}"`);

    this.content.push(
      `<prosody ${attrs.join(' ')}>${this.escapeText(text)}</prosody>`
    );
    return this;
  }

  phoneme(text, alphabet, pronunciation) {
    this.content.push(
      `<phoneme alphabet="${alphabet}" ph="${pronunciation}">${this.escapeText(text)}</phoneme>`
    );
    return this;
  }

  sayAs(text, interpretAs, format = null) {
    const formatAttr = format ? ` format="${format}"` : '';
    this.content.push(
      `<say-as interpret-as="${interpretAs}"${formatAttr}>${this.escapeText(text)}</say-as>`
    );
    return this;
  }

  paragraph(callback) {
    this.content.push('<p>');
    callback();
    this.content.push('</p>');
    return this;
  }

  sentence(callback) {
    this.content.push('<s>');
    callback();
    this.content.push('</s>');
    return this;
  }

  escapeText(text) {
    return text
      .replace(/&/g, '&amp;')
      .replace(/</g, '&lt;')
      .replace(/>/g, '&gt;')
      .replace(/"/g, '&quot;')
      .replace(/'/g, '&apos;');
  }

  toString() {
    return this.content.join('');
  }

  clear() {
    this.content = [];
    return this;
  }
}

// SSML Templates for common use cases
class SSMLTemplates {
  static meditation(content) {
    return new SSMLBuilder()
      .speak(() => {
        new SSMLBuilder()
          .prosody('Welcome to your mindfulness practice.', {
            rate: 'slow',
            pitch: 'low',
          })
          .pause('2s')
          .prosody(content, { rate: 'x-slow' })
          .pause('3s')
          .prosody('Take a moment to simply be present.', { rate: 'slow' });
      })
      .toString();
  }

  static lecture(title, content) {
    return new SSMLBuilder()
      .speak(() => {
        new SSMLBuilder()
          .paragraph(() => {
            new SSMLBuilder().emphasis(title, 'strong').pause('1s');
          })
          .paragraph(() => {
            new SSMLBuilder().prosody(content, { rate: 'medium' });
          });
      })
      .toString();
  }

  static conversation(content) {
    return new SSMLBuilder()
      .speak(() => {
        new SSMLBuilder().prosody(content, { rate: 'medium', pitch: 'medium' });
      })
      .toString();
  }

  static breathingExercise(steps) {
    const builder = new SSMLBuilder();

    return builder
      .speak(() => {
        builder
          .prosody("Let's begin the breathing exercise.", { rate: 'slow' })
          .pause('2s');

        steps.forEach((step, index) => {
          builder
            .prosody(`Step ${index + 1}.`, { rate: 'slow' })
            .pause('500ms')
            .prosody(step.instruction, { rate: 'slow' })
            .pause(step.duration || '3s');
        });

        builder
          .pause('2s')
          .prosody('Well done. Take a moment to notice how you feel.', {
            rate: 'slow',
          });
      })
      .toString();
  }

  static pronounciation(word, phonetic, explanation = null) {
    const builder = new SSMLBuilder();

    return builder
      .speak(() => {
        builder.text('The word ');

        if (explanation) {
          builder
            .phoneme(word, 'ipa', phonetic)
            .pause('500ms')
            .text(explanation);
        } else {
          builder
            .phoneme(word, 'ipa', phonetic)
            .pause('300ms')
            .text('is pronounced')
            .pause('300ms')
            .phoneme(word, 'ipa', phonetic);
        }
      })
      .toString();
  }
}

// Usage examples
const meditationSSML = SSMLTemplates.meditation(
  'Breathe in slowly through your nose, feeling your belly expand. Hold for three counts. Now breathe out slowly through your mouth.'
);

const lectureSSML = SSMLTemplates.lecture(
  'The Four Noble Truths',
  'The Four Noble Truths form the foundation of Buddhist teaching and provide a framework for understanding suffering and the path to liberation.'
);

const breathingSSML = SSMLTemplates.breathingExercise([
  {
    instruction: 'Breathe in slowly through your nose for four counts.',
    duration: '5s',
  },
  { instruction: 'Hold your breath for four counts.', duration: '5s' },
  {
    instruction: 'Breathe out slowly through your mouth for six counts.',
    duration: '7s',
  },
  { instruction: 'Rest and notice the stillness.', duration: '3s' },
]);

Audio Formats#

Format Selection and Optimization#

javascript
class AudioFormatManager {
  constructor() {
    this.formats = {
      opus: {
        mimeType: 'audio/ogg; codecs="opus"',
        extension: '.opus',
        quality: 'high',
        compression: 'excellent',
        streaming: true,
        browserSupport: ['chrome', 'firefox', 'edge'],
        recommendedFor: ['streaming', 'real-time', 'bandwidth-limited'],
      },
      mp3: {
        mimeType: 'audio/mpeg',
        extension: '.mp3',
        quality: 'good',
        compression: 'good',
        streaming: false,
        browserSupport: ['all'],
        recommendedFor: ['compatibility', 'downloads', 'legacy-systems'],
      },
      wav: {
        mimeType: 'audio/wav',
        extension: '.wav',
        quality: 'highest',
        compression: 'none',
        streaming: false,
        browserSupport: ['all'],
        recommendedFor: ['editing', 'archival', 'high-fidelity'],
      },
      webm: {
        mimeType: 'audio/webm; codecs="opus"',
        extension: '.webm',
        quality: 'high',
        compression: 'excellent',
        streaming: true,
        browserSupport: ['chrome', 'firefox'],
        recommendedFor: ['web', 'streaming'],
      },
    };

    this.qualityPresets = {
      low: { bitrate: 32000, sampleRate: 22050 },
      medium: { bitrate: 64000, sampleRate: 44100 },
      high: { bitrate: 128000, sampleRate: 44100 },
      highest: { bitrate: 256000, sampleRate: 48000 },
    };
  }

  detectBestFormat(capabilities = {}) {
    const userAgent = navigator.userAgent.toLowerCase();
    const supports = {
      opus: this.canPlayType('audio/ogg; codecs="opus"'),
      webm: this.canPlayType('audio/webm; codecs="opus"'),
      mp3: this.canPlayType('audio/mpeg'),
      wav: this.canPlayType('audio/wav'),
    };

    // Prefer formats in order of quality and compression
    const preferenceOrder = ['opus', 'webm', 'mp3', 'wav'];

    for (const format of preferenceOrder) {
      if (supports[format] === 'probably' || supports[format] === 'maybe') {
        return {
          format: format,
          confidence: supports[format],
          details: this.formats[format],
        };
      }
    }

    // Fallback to MP3 (universal support)
    return {
      format: 'mp3',
      confidence: 'fallback',
      details: this.formats.mp3,
    };
  }

  canPlayType(mimeType) {
    const audio = new Audio();
    return audio.canPlayType(mimeType);
  }

  selectQuality(bandwidth, deviceType = 'desktop') {
    const qualityMap = {
      // Bandwidth in kbps
      0: 'low', // < 100 kbps
      100: 'medium', // 100-500 kbps
      500: 'high', // 500-1000 kbps
      1000: 'highest', // > 1000 kbps
    };

    const deviceAdjustment = {
      mobile: -1, // Lower quality for mobile
      tablet: 0, // Standard quality
      desktop: 1, // Higher quality for desktop
    };

    let targetQuality = 'medium';
    for (const [minBandwidth, quality] of Object.entries(qualityMap)) {
      if (bandwidth >= parseInt(minBandwidth)) {
        targetQuality = quality;
      }
    }

    // Adjust based on device type
    const adjustment = deviceAdjustment[deviceType] || 0;
    const qualities = Object.keys(this.qualityPresets);
    const currentIndex = qualities.indexOf(targetQuality);
    const newIndex = Math.max(
      0,
      Math.min(qualities.length - 1, currentIndex + adjustment)
    );

    return qualities[newIndex];
  }

  getOptimalSettings(format, quality, options = {}) {
    const baseSettings = {
      format: format,
      ...this.qualityPresets[quality],
    };

    // Content-specific optimizations
    if (options.contentType) {
      switch (options.contentType) {
        case 'meditation':
          baseSettings.bitrate = Math.min(baseSettings.bitrate, 96000); // Lower bitrate for calm content
          break;
        case 'lecture':
          baseSettings.sampleRate = Math.max(baseSettings.sampleRate, 44100); // Higher fidelity for speech
          break;
        case 'music':
          baseSettings.bitrate = Math.max(baseSettings.bitrate, 128000); // Higher bitrate for music
          break;
      }
    }

    // Streaming optimizations
    if (options.streaming) {
      baseSettings.streamingChunkSize = 4096;
      baseSettings.streamingBuffer = 2; // seconds
    }

    return baseSettings;
  }
}

// Usage
const formatManager = new AudioFormatManager();

// Auto-detect best format
const bestFormat = formatManager.detectBestFormat();
console.log('Best supported format:', bestFormat.format);

// Select quality based on connection
const quality = formatManager.selectQuality(800, 'desktop'); // 800 kbps, desktop
console.log('Selected quality:', quality);

// Get optimal settings
const settings = formatManager.getOptimalSettings(bestFormat.format, quality, {
  contentType: 'meditation',
  streaming: true,
});

// Use settings for TTS request
const audioRequest = {
  voice_id: 'calm_female_01',
  ssml: '<speak>Your meditation text here...</speak>',
  format: settings.format,
  bitrate: settings.bitrate,
  sample_rate: settings.sampleRate,
  stream: true,
};

Real-time Streaming#

WebSocket Audio Streaming#

javascript
class WebSocketAudioStreamer {
  constructor(apiToken) {
    this.apiToken = apiToken;
    this.ws = null;
    this.audioContext = null;
    this.audioQueue = [];
    this.isPlaying = false;
    this.currentSource = null;
  }

  async connect() {
    return new Promise((resolve, reject) => {
      this.ws = new WebSocket('wss://api.lilith.app/audio/stream');

      this.ws.onopen = async () => {
        // Initialize audio context
        this.audioContext = new (
          window.AudioContext || window.webkitAudioContext
        )();

        // Authenticate
        this.ws.send(
          JSON.stringify({
            type: 'auth',
            token: this.apiToken,
          })
        );

        resolve();
      };

      this.ws.onmessage = (event) => {
        this.handleMessage(event);
      };

      this.ws.onerror = (error) => {
        reject(error);
      };

      this.ws.onclose = () => {
        console.log('WebSocket audio stream closed');
        this.cleanup();
      };
    });
  }

  async handleMessage(event) {
    if (typeof event.data === 'string') {
      const message = JSON.parse(event.data);
      this.handleControlMessage(message);
    } else {
      // Binary audio data
      await this.handleAudioData(event.data);
    }
  }

  handleControlMessage(message) {
    switch (message.type) {
      case 'stream_start':
        console.log('Audio stream started');
        break;
      case 'stream_end':
        console.log('Audio stream ended');
        this.finishPlayback();
        break;
      case 'error':
        console.error('Stream error:', message.error);
        break;
    }
  }

  async handleAudioData(audioData) {
    try {
      const arrayBuffer = await audioData.arrayBuffer();
      const audioBuffer = await this.audioContext.decodeAudioData(arrayBuffer);

      this.audioQueue.push(audioBuffer);

      if (!this.isPlaying) {
        this.startPlayback();
      }
    } catch (error) {
      console.error('Error processing audio data:', error);
    }
  }

  startPlayback() {
    if (this.audioQueue.length === 0) {
      this.isPlaying = false;
      return;
    }

    this.isPlaying = true;
    this.playNextBuffer();
  }

  playNextBuffer() {
    if (this.audioQueue.length === 0) {
      this.isPlaying = false;
      return;
    }

    const buffer = this.audioQueue.shift();
    this.currentSource = this.audioContext.createBufferSource();
    this.currentSource.buffer = buffer;

    // Add gain control for smooth transitions
    const gainNode = this.audioContext.createGain();
    this.currentSource.connect(gainNode);
    gainNode.connect(this.audioContext.destination);

    this.currentSource.onended = () => {
      this.playNextBuffer();
    };

    this.currentSource.start();
  }

  streamText(text, voiceId, options = {}) {
    if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {
      throw new Error('WebSocket not connected');
    }

    this.ws.send(
      JSON.stringify({
        type: 'synthesize',
        data: {
          voice_id: voiceId,
          text: text,
          format: 'opus',
          stream: true,
          ...options,
        },
      })
    );
  }

  pause() {
    if (this.currentSource) {
      this.currentSource.stop();
      this.currentSource = null;
    }
    this.isPlaying = false;
  }

  resume() {
    if (!this.isPlaying && this.audioQueue.length > 0) {
      this.startPlayback();
    }
  }

  finishPlayback() {
    // Play remaining queued audio
    if (this.audioQueue.length > 0 && !this.isPlaying) {
      this.startPlayback();
    }
  }

  cleanup() {
    this.pause();
    this.audioQueue = [];

    if (this.audioContext) {
      this.audioContext.close();
      this.audioContext = null;
    }
  }

  disconnect() {
    if (this.ws) {
      this.ws.close();
      this.ws = null;
    }
    this.cleanup();
  }
}

// Usage example
const streamer = new WebSocketAudioStreamer(apiToken);

await streamer.connect();

// Stream long-form content with real-time playback
streamer.streamText(
  'This is a long meditation guide that will be streamed in real-time as it is being synthesized, providing a seamless listening experience.',
  'calm_female_01',
  {
    speed: 0.8,
    pitch: -2.0,
  }
);

// Control playback
document.getElementById('pauseBtn').onclick = () => streamer.pause();
document.getElementById('resumeBtn').onclick = () => streamer.resume();

Due to the extensive nature of this documentation system, I need to continue with the remaining files. Let me complete a few more critical files and then provide a summary.

<function_calls> <invoke name="TodoWrite"> <parameter name="todos">[{"content": "Create comprehensive documentation directory structure with all required folders", "status": "completed", "activeForm": "Creating comprehensive documentation directory structure"}, {"content": "Set up Stoplight Studio configuration files and enhanced OpenAPI spec", "status": "completed", "activeForm": "Setting up Stoplight Studio configuration"}, {"content": "Create documentation package.json with all required dependencies and scripts", "status": "completed", "activeForm": "Creating documentation package configuration"}, {"content": "Configure portal settings with branding, navigation, and interactive features", "status": "completed", "activeForm": "Configuring portal settings and branding"}, {"content": "Write comprehensive guides and getting started documentation", "status": "completed", "activeForm": "Writing comprehensive guides and documentation"}, {"content": "Create detailed examples and code samples for all supported languages", "status": "in_progress", "activeForm": "Creating examples and code samples"}, {"content": "Build API reference documentation and SDK integration guides", "status": "pending", "activeForm": "Building API reference and SDK guides"}, {"content": "Set up publishing pipeline with GitHub Actions and validation tools", "status": "pending", "activeForm": "Setting up publishing pipeline"}, {"content": "Configure SEO, analytics, and performance optimization features", "status": "pending", "activeForm": "Configuring SEO and analytics features"}, {"content": "Create version management and synchronization configuration", "status": "pending", "activeForm": "Creating version management configuration"}] </parameter> </invoke> </function_calls>

Advanced Background Audio#

The AdvancedAudioEngine orchestrates layered meditation soundscapes, binaural beats, and adaptive processing on the client.

  • Layered soundscapes β€” call loadSoundscape and playSoundscape to manage multiple looping ambience stems with automatic fades.
  • State-aware modulation β€” annotate each SoundLayer with a reactivity map describing how calmness, focus, or energy should scale that layer. Feeding live biometrics into updateUserState rebalances volumes and retunes the binaural beat generator.
  • Testable injection β€” non-browser environments can supply mocks through AdvancedAudioEngine.getInstance(config, dependencies); our Vitest suite exercises the entire pipeline without a real AudioContext.

Example adaptive configuration:

ts
import { AdvancedAudioEngine } from '@/lib/advanced-audio-engine';

const engine = AdvancedAudioEngine.getInstance();

await engine.loadSoundscape({
  id: 'forest-calm',
  name: 'Forest Calmness',
  adaptive: true,
  stateMapping: { calmness: 0.4, focus: 0.6, energy: 0.5 },
  layers: [
    {
      url: '/audio/forest/base.mp3',
      volume: 0.6,
      loop: true,
      fadeIn: 4,
      reactivity: { calmness: 1 },
    },
    {
      url: '/audio/forest/detail.mp3',
      volume: 0.3,
      loop: true,
      fadeIn: 6,
      reactivity: { focus: 0.6, energy: 0.4 },
    },
  ],
});

engine.playSoundscape('forest-calm');

// Apply live feedback based on biometrics
engine.updateUserState({ calmness: 0.85, focus: 0.3, energy: 0.4 });

Run the targeted coverage via:

bash
cd web
timeout -k 5 120s npx vitest run src/lib/__tests__/advanced-audio-engine.test.ts

The test harness validates spatial scene wiring, fade curves, adaptive reactivity, and binaural retuning with fully mocked audio primitives.