Eve library

Emotion Detection Provider — V1-P2-0215

Guide

Revision 1

On this page

Decision#

Voice emotion: ONNX wav2vec2-emotion (6-class arousal/valence). Image emotion: ONNX HSEmotion (FER+ 7-class). Both run locally through the shared ML runtime; cloud APIs (AWS Comprehend Sentiment, Affectiva Affdex) are kept on hand for the long-tail "research-grade affect" flow but not used by default.

Emotion classification is inherently imprecise; we ship 6 / 7 discrete classes plus an arousal-valence continuous pair so callers can choose the granularity they need. We deliberately do NOT ship a "depression detector" or any clinical-grade affect model — the failure-mode cost is too high for what's a probabilistic signal.


Constraints#

  1. Privacy. Same as speaker-id (V1-P2-0214) — emotion signals are sensitive PII; default path is local.
  2. Latency. Voice emotion: ≤ 80 ms per 1 s of audio. Image emotion: ≤ 30 ms per face crop on WebGPU.
  3. Honest about uncertainty. Output must include per-class probability, not a single hard label.

Options#

Voice — ONNX wav2vec2-emotion (chosen)#

  • HuggingFace audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim exported to ONNX. Outputs three continuous dims (arousal / dominance / valence) + 6-class softmax (neutral / happy / angry / sad / fearful / disgusted).
  • ~310 MB FP16. We ship the INT8 quantised version (~80 MB) by default.
  • Latency: ~50 ms per 1 s on WebGPU; ~140 ms WASM SIMD.

Image — HSEmotion ONNX (chosen)#

  • hsemotion repository ships a small (~2 MB) MobileNet-V2 backbone trained on AffectNet + FER+. 7-class output (neutral / happy / surprise / sad / anger / disgust / fear / contempt).
  • Latency: ~5 ms per face crop on WebGPU.
  • Pairs naturally with the face-detection step from the object-detection provider (V1-P2-0212).

Cloud — AWS Comprehend / Affectiva (kept, not default)#

  • More classes, fine-grained intensity, in some cases (Affectiva) facial action unit (AU) decomposition.
  • Useful when a research / enterprise flow needs auditable third-party affect labels.

Clinical-grade affect — rejected#

  • Depression / bipolar / suicidal-ideation classification from voice or text is well-established as unreliable in the open literature. We refuse to ship this.

Output schema#

ts
export interface VoiceEmotionResult {
  readonly arousal: number; // [-1, 1]
  readonly valence: number; // [-1, 1]
  readonly dominance: number; // [-1, 1]
  readonly classes: Readonly<
    Record<
      'neutral' | 'happy' | 'angry' | 'sad' | 'fearful' | 'disgusted',
      number /* probability in [0,1] */
    >
  >;
}

export interface ImageEmotionResult {
  readonly classes: Readonly<
    Record<
      | 'neutral'
      | 'happy'
      | 'surprise'
      | 'sad'
      | 'anger'
      | 'disgust'
      | 'fear'
      | 'contempt',
      number
    >
  >;
}

Action items#

  • Pull audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim and export to ONNX; quantise to INT8.
  • Pull hsemotion-onnx (MobileNet-V2 backbone); check FP16 + INT8 variants into the registry.
  • Wire libs/shared/src/ml/emotion.ts namespace with voiceEmotion(audio) and faceEmotion(faceCrop) entrypoints.
  • Add disclaimer flag to the API surface so callers programmatically receive a "probabilistic signal — do not use for clinical decisions" note.

References: