# Audio Processing Pipeline

This document provides comprehensive documentation for the Lilith Media Service
Audio Processing Pipeline, including transcoding, normalization, silence
detection/removal, waveform generation, and audio fingerprinting.

## Overview

The Audio Processing Pipeline provides:

- **Audio Transcoding**: Convert between formats (MP3, AAC, Opus, WAV, FLAC,
  OGG)
- **Audio Normalization**: LUFS-based loudness normalization with platform
  presets
- **Silence Processing**: Detection, removal, trimming, and splitting at silence
- **Waveform Generation**: Visual waveform images and JSON peak data
- **Audio Fingerprinting**: Chromaprint-based fingerprinting for duplicate
  detection

---

## Table of Contents

1. [Audio Transcoding](#audio-transcoding)
2. [Audio Normalization](#audio-normalization)
3. [Silence Processing](#silence-processing)
4. [Waveform Generation](#waveform-generation)
5. [Audio Fingerprinting](#audio-fingerprinting)

---

## Audio Transcoding

### Supported Formats

| Format | Codec        | Extension | Default Bitrate | Use Case                           |
| ------ | ------------ | --------- | --------------- | ---------------------------------- |
| `mp3`  | libmp3lame   | .mp3      | 192 kbps        | General audio, broad compatibility |
| `aac`  | aac (LC-AAC) | .m4a      | 128 kbps        | Mobile streaming, Apple ecosystem  |
| `opus` | libopus      | .opus     | 96 kbps         | Voice, VoIP, web streaming         |
| `wav`  | pcm_s16le    | .wav      | Lossless        | Editing, archival                  |
| `flac` | flac         | .flac     | Lossless        | Lossless archival                  |
| `ogg`  | libvorbis    | .ogg      | 128 kbps        | Open source, web audio             |

### Transcode Options

```typescript
interface AudioTranscodeOptions {
  format: 'mp3' | 'aac' | 'opus' | 'wav' | 'flac' | 'ogg';
  bitrate?: number; // kbps (e.g., 128, 192, 256, 320)
  sampleRate?: number; // Hz (e.g., 44100, 48000)
  channels?: number; // 1 = mono, 2 = stereo
  codec?: string; // Override default codec
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number; // ms (default: 30 minutes)
}
```

### Transcoding Presets

| Preset          | Format | Bitrate  | Sample Rate | Channels | Use Case          |
| --------------- | ------ | -------- | ----------- | -------- | ----------------- |
| `mp3_high`      | mp3    | 320 kbps | 44.1 kHz    | stereo   | Music archival    |
| `mp3_standard`  | mp3    | 192 kbps | 44.1 kHz    | stereo   | General use       |
| `mp3_voice`     | mp3    | 96 kbps  | 44.1 kHz    | mono     | Podcasts, voice   |
| `aac_streaming` | aac    | 128 kbps | 44.1 kHz    | stereo   | Mobile streaming  |
| `aac_high`      | aac    | 256 kbps | 48 kHz      | stereo   | High-quality AAC  |
| `opus_voice`    | opus   | 32 kbps  | 48 kHz      | mono     | Voice chat, VoIP  |
| `opus_music`    | opus   | 128 kbps | 48 kHz      | stereo   | Music streaming   |
| `wav_lossless`  | wav    | Lossless | 44.1 kHz    | stereo   | Editing source    |
| `flac_lossless` | flac   | Lossless | 44.1 kHz    | stereo   | Lossless archival |

### Usage Example

```typescript
import {
  transcodeAudio,
  transcodeAudioBuffer,
  batchTranscode,
  TRANSCODE_PRESETS,
} from './processing/audio-transcoder.js';

// Single file transcoding
const result = await transcodeAudio(
  '/path/to/input.wav',
  '/path/to/output.mp3',
  {
    format: 'mp3',
    bitrate: 320,
    sampleRate: 44100,
    onProgress: (progress) => {
      console.log(`Progress: ${progress.percent}%`);
    },
  }
);

console.log(
  `Transcoded: ${result.outputFormat} at ${result.outputBitrate}kbps`
);

// Using presets
const voiceResult = await transcodeAudio(
  '/path/to/voice.wav',
  '/path/to/voice.mp3',
  TRANSCODE_PRESETS.mp3_voice
);

// Buffer transcoding
const { outputBuffer, result } = await transcodeAudioBuffer(
  inputBuffer,
  '.wav',
  { format: 'aac', bitrate: 128 }
);

// Batch transcoding
const files = [
  { inputPath: 'track1.wav', outputPath: 'track1.mp3' },
  { inputPath: 'track2.wav', outputPath: 'track2.mp3' },
];

const results = await batchTranscode(files, { format: 'mp3', bitrate: 192 });
```

---

## Audio Normalization

### Overview

Audio normalization adjusts loudness to target levels using the EBU R128
standard (LUFS measurement). This ensures consistent playback volume across
different audio files.

### Normalization Options

```typescript
interface NormalizationOptions {
  targetLoudness?: number; // LUFS (default: -16)
  truePeak?: number; // dBTP (default: -1.5)
  loudnessRange?: number; // LU (default: 11)
  dualPass?: boolean; // Two-pass for accuracy (default: true)
  linear?: boolean; // No compression (default: false)
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number;
}
```

### Platform Presets

| Preset      | Target LUFS | True Peak | LRA   | Description             |
| ----------- | ----------- | --------- | ----- | ----------------------- |
| `spotify`   | -14 LUFS    | -1 dBTP   | -     | Spotify recommendation  |
| `youtube`   | -14 LUFS    | -1 dBTP   | -     | YouTube recommendation  |
| `apple`     | -16 LUFS    | -1 dBTP   | -     | Apple Music / iTunes    |
| `podcast`   | -16 LUFS    | -1.5 dBTP | 8 LU  | Podcast standard        |
| `voice`     | -16 LUFS    | -3 dBTP   | 5 LU  | Voice/speech (tight DR) |
| `broadcast` | -23 LUFS    | -1 dBTP   | 15 LU | EBU R128 broadcast      |
| `peak_only` | -16 LUFS    | -1 dBTP   | -     | Simple peak limiting    |

### Loudness Analysis

```typescript
interface LoudnessInfo {
  integratedLoudness: number; // LUFS
  truePeak: number; // dBTP
  loudnessRange: number; // LU
  threshold: number;
  targetOffset?: number;
}
```

### Usage Example

```typescript
import {
  normalizeAudio,
  analyzeLoudness,
  needsNormalization,
  applyLimiter,
  applyCompressor,
  NORMALIZATION_PRESETS,
} from './processing/audio-normalizer.js';

// Analyze loudness
const loudness = await analyzeLoudness('/path/to/audio.mp3');
console.log(`Current loudness: ${loudness.integratedLoudness} LUFS`);
console.log(`True peak: ${loudness.truePeak} dBTP`);

// Check if normalization is needed
const check = await needsNormalization('/path/to/audio.mp3', -16);
if (check.needsNormalization) {
  console.log(`Needs adjustment: ${check.delta} dB`);
}

// Normalize audio
const result = await normalizeAudio(
  '/path/to/input.mp3',
  '/path/to/output.wav',
  {
    targetLoudness: -14,
    truePeak: -1,
    dualPass: true,
  }
);

console.log(`Gain applied: ${result.gainAdjustment} dB`);
console.log(
  `Output loudness: ${result.outputLoudness?.integratedLoudness} LUFS`
);

// Using platform presets
const spotifyResult = await normalizeAudio(
  '/path/to/input.mp3',
  '/path/to/spotify-ready.wav',
  NORMALIZATION_PRESETS.spotify
);

// Apply limiter
await applyLimiter('/path/to/input.wav', '/path/to/limited.wav', {
  threshold: -1, // dB
  attack: 5, // ms
  release: 50, // ms
  makeup: 0, // dB
});

// Apply compressor
await applyCompressor('/path/to/input.wav', '/path/to/compressed.wav', {
  threshold: -20, // dB
  ratio: 4, // 4:1 compression
  attack: 20, // ms
  release: 250, // ms
  knee: 2.82843, // dB
});
```

---

## Silence Processing

### Silence Detection

Detect silence segments in audio files based on noise threshold and minimum
duration.

```typescript
interface SilenceDetectionResult {
  segments: SilenceSegment[];
  totalSilenceDuration: number;
  audioDuration: number;
  silencePercentage: number;
}

interface SilenceSegment {
  start: number; // seconds
  end: number; // seconds
  duration: number; // seconds
}
```

### Silence Removal Options

```typescript
interface SilenceRemovalOptions {
  noiseThreshold?: number; // dB (default: -50)
  minDuration?: number; // seconds (default: 0.5)
  padding?: number; // Keep around speech (default: 0.1s)
  maxSilence?: number; // Max silence between speech (default: 0.5s)
  trimStart?: boolean; // Remove from start only
  trimEnd?: boolean; // Remove from end only
}
```

### Content Presets

| Preset          | Threshold | Min Duration | Padding | Max Silence | Use Case            |
| --------------- | --------- | ------------ | ------- | ----------- | ------------------- |
| `podcast`       | -40 dB    | 0.3s         | 0.05s   | 0.3s        | Aggressive removal  |
| `music`         | -60 dB    | 2.0s         | 0.2s    | 1.0s        | Preserve dynamics   |
| `voice`         | -45 dB    | 0.5s         | 0.1s    | 0.5s        | Moderate removal    |
| `audiobook`     | -45 dB    | 1.0s         | 0.15s   | 0.8s        | Keep natural pauses |
| `transcription` | -35 dB    | 0.2s         | 0.02s   | 0.2s        | Prep for STT        |

### Usage Example

```typescript
import {
  detectSilence,
  removeSilence,
  trimSilence,
  compressSilence,
  splitAtSilence,
  getSilenceStats,
  SILENCE_PRESETS,
} from './processing/silence-processor.js';

// Detect silence segments
const silence = await detectSilence('/path/to/audio.mp3', {
  noiseThreshold: -50,
  minDuration: 0.5,
});

console.log(`Found ${silence.segments.length} silence segments`);
console.log(`${silence.silencePercentage.toFixed(1)}% of audio is silence`);

// Get detailed statistics
const stats = getSilenceStats(silence);
console.log(`Average silence: ${stats.averageDuration.toFixed(2)}s`);
console.log(`Longest silence: ${stats.maxDuration.toFixed(2)}s`);

// Remove all silence
const result = await removeSilence(
  '/path/to/input.mp3',
  '/path/to/no-silence.wav',
  { noiseThreshold: -45, minDuration: 0.3 }
);

console.log(`Removed ${result.durationRemoved.toFixed(2)}s of silence`);

// Trim silence from start/end only
await trimSilence('/path/to/input.mp3', '/path/to/trimmed.wav', {
  padding: 0.1,
});

// Compress pauses (keep shorter silence)
await compressSilence('/path/to/input.mp3', '/path/to/compressed.wav', {
  maxSilence: 0.5,
});

// Split at silence points
const { segments, silenceInfo } = await splitAtSilence(
  '/path/to/audio.mp3',
  '/path/to/output/',
  {
    ...SILENCE_PRESETS.podcast,
    outputFormat: 'wav',
    minSegmentDuration: 2.0,
  }
);

console.log(`Created ${segments.length} segments`);
segments.forEach((seg, i) => {
  console.log(`Segment ${i + 1}: ${seg.duration.toFixed(2)}s`);
});
```

---

## Waveform Generation

### Overview

Generate visual waveform representations as images (PNG) or data arrays for
frontend rendering.

### Waveform Options

```typescript
interface WaveformOptions {
  samples?: number; // Number of bars (default: 800)
  width?: number; // Image width px (default: 1800)
  height?: number; // Image height px (default: 140)
  color?: string; // Hex color (default: '#5865F2')
  backgroundColor?: string; // Hex or 'transparent'
  mirror?: boolean; // Mirrored waveform (default: true)
  scale?: 'lin' | 'log' | 'sqrt'; // Scale type (default: 'sqrt')
}
```

### Waveform Presets

| Preset      | Samples | Dimensions | Scale  | Use Case         |
| ----------- | ------- | ---------- | ------ | ---------------- |
| `detailed`  | 2000    | 3600x280   | linear | Full detail view |
| `player`    | 800     | 1800x140   | sqrt   | Audio player UI  |
| `thumbnail` | 200     | 400x60     | sqrt   | List thumbnails  |
| `mobile`    | 100     | 300x40     | sqrt   | Mobile devices   |

### Waveform Data Format

```typescript
interface WaveformDataResult {
  peaks: number[]; // Amplitude values (0.0 to 1.0)
  duration: number; // Audio duration in seconds
  sampleRate: number; // Samples per second
  channels: number; // Channel count
  bitsPerSample: number;
}
```

### Usage Example

```typescript
import {
  generateWaveformImage,
  generateWaveformData,
  generateWaveform,
  peaksToSvgPath,
  downsamplePeaks,
  WAVEFORM_PRESETS,
} from './processing/waveform-generator.js';

// Generate waveform image
const imageResult = await generateWaveformImage(
  '/path/to/audio.mp3',
  '/path/to/waveform.png',
  {
    width: 1800,
    height: 140,
    color: '#5865F2',
    backgroundColor: 'transparent',
    mirror: true,
    scale: 'sqrt',
  }
);

console.log(`Generated waveform: ${imageResult.width}x${imageResult.height}`);

// Generate waveform data (for frontend)
const data = await generateWaveformData('/path/to/audio.mp3', {
  samples: 800,
  scale: 'sqrt',
});

console.log(`${data.peaks.length} samples, ${data.duration}s duration`);

// Generate both image and data
const { image, data: waveformData } = await generateWaveform(
  '/path/to/audio.mp3',
  '/path/to/waveform.png',
  WAVEFORM_PRESETS.player
);

// Convert peaks to SVG path
const svgPath = peaksToSvgPath(data.peaks, 800, 100, {
  mirror: true,
  smooth: true,
});

// Use in SVG:
// <path d="${svgPath}" fill="#5865F2" />

// Downsample for different views
const mobilePeaks = downsamplePeaks(data.peaks, 100);

// Generate from buffer
const { imageBuffer, result } = await generateWaveformImageFromBuffer(
  audioBuffer,
  '.mp3',
  WAVEFORM_PRESETS.thumbnail
);
```

---

## Audio Fingerprinting

### Overview

Generate audio fingerprints using Chromaprint (fpcalc) for audio identification,
duplicate detection, and integration with music recognition services like
AcoustID.

### Prerequisites

Requires Chromaprint's `fpcalc` utility:

- macOS: `brew install chromaprint`
- Ubuntu: `apt-get install libchromaprint-tools`
- Windows: Download from https://acoustid.org/chromaprint

### Fingerprint Result

```typescript
interface FingerprintResult {
  fingerprint: string; // Raw fingerprint
  duration: number; // Duration analyzed
  fingerprintBase64: string; // Base64 encoded
  fingerprintHash: string; // SHA-256 hash
  algorithm: number; // Algorithm version
}
```

### Fingerprint Options

```typescript
interface FingerprintOptions {
  maxDuration?: number; // Max seconds to analyze (default: 120)
  startOffset?: number; // Start offset in seconds (default: 0)
  algorithm?: number; // Algorithm version (default: 2)
  overlap?: number; // Overlap factor (default: 0)
}
```

### Usage Example

```typescript
import {
  generateFingerprint,
  compareFingerprints,
  compareRawFingerprints,
  areDuplicates,
  findDuplicates,
  batchFingerprint,
  prepareForAcoustId,
  isFpcalcAvailable,
} from './processing/audio-fingerprint.js';

// Check if fpcalc is available
if (await isFpcalcAvailable()) {
  console.log('Chromaprint is installed');
}

// Generate fingerprint
const fingerprint = await generateFingerprint('/path/to/audio.mp3', {
  maxDuration: 120,
  algorithm: 2,
});

console.log(`Fingerprint hash: ${fingerprint.fingerprintHash}`);
console.log(`Duration analyzed: ${fingerprint.duration}s`);

// Compare two fingerprints
const fp1 = await generateFingerprint('/path/to/song1.mp3');
const fp2 = await generateFingerprint('/path/to/song2.mp3');

const comparison = compareFingerprints(fp1, fp2, {
  threshold: 0.9,
  method: 'hash',
});

console.log(`Similarity: ${(comparison.similarity * 100).toFixed(1)}%`);
console.log(`Match: ${comparison.match}`);

// Check if files are duplicates
const duplicateCheck = await areDuplicates(
  '/path/to/file1.mp3',
  '/path/to/file2.mp3',
  { threshold: 0.85 }
);

if (duplicateCheck.isDuplicate) {
  console.log(
    `Duplicates! Similarity: ${(duplicateCheck.similarity * 100).toFixed(1)}%`
  );
}

// Batch fingerprint multiple files
const fingerprints = await batchFingerprint(
  ['song1.mp3', 'song2.mp3', 'song3.mp3'],
  {
    maxDuration: 120,
    concurrency: 4,
    onProgress: (i, total, file) => {
      console.log(`Processing ${i}/${total}: ${file}`);
    },
  }
);

// Find duplicates in collection
const duplicates = await findDuplicates(
  ['song1.mp3', 'song2.mp3', 'song3.mp3', 'song4.mp3'],
  {
    threshold: 0.85,
    onProgress: (current, total) => {
      console.log(`Comparing: ${current}/${total}`);
    },
  }
);

duplicates.forEach((dup) => {
  console.log(`Duplicate pair: ${dup.files[0]} <-> ${dup.files[1]}`);
  console.log(`Similarity: ${(dup.similarity * 100).toFixed(1)}%`);
});

// Prepare for AcoustID lookup
const acoustId = prepareForAcoustId(fingerprint);
console.log(
  `AcoustID fingerprint: ${acoustId.acoustIdFingerprint.substring(0, 50)}...`
);
console.log(`Duration: ${acoustId.acoustIdDuration}s`);
```

---

## Processing Pipeline Integration

### Requesting Processing Jobs

Use the Asset API to request audio processing:

```bash
curl -X POST "https://api.lilith.app/v1/assets/asset-abc123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "transcode",
    "config": {
      "format": "mp3",
      "bitrate": 192,
      "normalize": true,
      "targetLoudness": -14
    }
  }'
```

### Processing Job Types

| Type                  | Description                     |
| --------------------- | ------------------------------- |
| `transcode`           | Convert audio format            |
| `normalization`       | Apply loudness normalization    |
| `silence_removal`     | Remove or compress silence      |
| `waveform`            | Generate waveform visualization |
| `fingerprint`         | Generate audio fingerprint      |
| `metadata_extraction` | Extract detailed audio metadata |

---

## SDK Integration

### TypeScript/JavaScript

```typescript
import { LilithClient } from '@lilith/sdk';

const client = new LilithClient({
  apiKey: process.env.LILITH_API_KEY,
});

// Request transcoding
const job = await client.media.requestProcessing('asset-abc123', {
  type: 'transcode',
  config: {
    format: 'mp3',
    bitrate: 320,
    normalize: true,
    targetLoudness: -14,
  },
});

// Monitor job progress
const status = await client.media.getProcessingJob(job.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);

// Get waveform data
const waveform = await client.media.getWaveform('asset-abc123', {
  samples: 800,
});
```

---

## Related Documentation

- [Asset Management API](./asset-api.md) - Asset upload and management
- [Video Processing](./video-processing.md) - Video transcoding and HLS
  packaging
- [Transcript API](./transcript-api.md) - Transcript management
- [CDN Setup](./cdn-setup.md) - CloudFront CDN configuration

---

## Questions?

- **Technical Support**: api-support@lilith.app
- **Documentation**: [GitHub Issues](https://github.com/lilith/platform/issues)
