# Emotion Detection Provider — V1-P2-0215

## Decision

**Voice emotion: ONNX wav2vec2-emotion (6-class arousal/valence). Image emotion:
ONNX HSEmotion (FER+ 7-class). Both run locally through the shared ML runtime;
cloud APIs (AWS Comprehend Sentiment, Affectiva Affdex) are kept on hand for the
long-tail "research-grade affect" flow but not used by default.**

Emotion classification is inherently imprecise; we ship 6 / 7 discrete classes
plus an arousal-valence continuous pair so callers can choose the granularity
they need. We deliberately do NOT ship a "depression detector" or any
clinical-grade affect model — the failure-mode cost is too high for what's a
probabilistic signal.

---

## Constraints

1. **Privacy.** Same as speaker-id (V1-P2-0214) — emotion signals are sensitive
   PII; default path is local.
2. **Latency.** Voice emotion: ≤ 80 ms per 1 s of audio. Image emotion: ≤ 30 ms
   per face crop on WebGPU.
3. **Honest about uncertainty.** Output must include per-class probability, not
   a single hard label.

---

## Options

### Voice — ONNX wav2vec2-emotion (chosen)

- HuggingFace `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` exported
  to ONNX. Outputs three continuous dims (arousal / dominance / valence) +
  6-class softmax (neutral / happy / angry / sad / fearful / disgusted).
- ~310 MB FP16. We ship the INT8 quantised version (~80 MB) by default.
- Latency: ~50 ms per 1 s on WebGPU; ~140 ms WASM SIMD.

### Image — HSEmotion ONNX (chosen)

- `hsemotion` repository ships a small (~2 MB) MobileNet-V2 backbone trained on
  AffectNet + FER+. 7-class output (neutral / happy / surprise / sad / anger /
  disgust / fear / contempt).
- Latency: ~5 ms per face crop on WebGPU.
- Pairs naturally with the face-detection step from the object-detection
  provider (V1-P2-0212).

### Cloud — AWS Comprehend / Affectiva (kept, not default)

- More classes, fine-grained intensity, in some cases (Affectiva) facial action
  unit (AU) decomposition.
- Useful when a research / enterprise flow needs auditable third-party affect
  labels.

### Clinical-grade affect — rejected

- Depression / bipolar / suicidal-ideation classification from voice or text is
  well-established as unreliable in the open literature. We refuse to ship this.

---

## Output schema

```ts
export interface VoiceEmotionResult {
  readonly arousal: number; // [-1, 1]
  readonly valence: number; // [-1, 1]
  readonly dominance: number; // [-1, 1]
  readonly classes: Readonly<
    Record<
      'neutral' | 'happy' | 'angry' | 'sad' | 'fearful' | 'disgusted',
      number /* probability in [0,1] */
    >
  >;
}

export interface ImageEmotionResult {
  readonly classes: Readonly<
    Record<
      | 'neutral'
      | 'happy'
      | 'surprise'
      | 'sad'
      | 'anger'
      | 'disgust'
      | 'fear'
      | 'contempt',
      number
    >
  >;
}
```

---

## Action items

- [ ] Pull `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` and export to
      ONNX; quantise to INT8.
- [ ] Pull `hsemotion-onnx` (MobileNet-V2 backbone); check FP16 + INT8 variants
      into the registry.
- [ ] Wire `libs/shared/src/ml/emotion.ts` namespace with `voiceEmotion(audio)`
      and `faceEmotion(faceCrop)` entrypoints.
- [ ] Add `disclaimer` flag to the API surface so callers programmatically
      receive a "probabilistic signal — do not use for clinical decisions" note.

References:

- [audeering wav2vec2 emotion](https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim)
- [HSEmotion](https://github.com/HSE-asavchenko/face-emotion-recognition)
- [AffectNet](http://mohammadmahoor.com/affectnet/)
