Eve library

Hard delivery conditions (2026-09-10)

Guide

Revision 1

On this page

Evidence for 182.C.18.08, which names nine: fast and slow speech, whisper, shouting, laughter, singing, emotional delivery, background music, accent variation, and overlapping speech. Those nine phrases expand to ten conditions in code, because "fast/slow speech" is two and they behave differently.

Four of the ten were measured. Six were not, and could not be.

Why six could not be built#

The four that were built are TRANSFORMATIONS of a signal — a tempo change, a mix, an offset copy — and each produces audio that genuinely has the property. The six are different phonation or different speakers:

Condition Why a filter will not do it
whisper whispered speech is unvoiced; filtering a voice gives a quiet voice
shouting shouting changes formants and phonation, not only level
laughter not speech; cannot come out of a speech clip
singing pitch-shifting speech to a melody is not singing
emotional delivery one Kokoro voice is cached and it has no emotion control
accent variation an accent is another person

A fixture that merely looks like the condition proves the detector reads the filter. All six need recorded speech — the 182.C.33 corpus.

What the four measured, and the two holes they found#

Onset errors are against the KNOWN onsets scaled by the tempo factor, so a correct aligner on a 1.25x clip reads near zero rather than 25% late. Onsets carry the 182.C.18.02 calibration of −48.288 ms.

Fixture max onset error stretch slope minGop est. SNR
normal 32.9 ms 0.994 0.00 81.3 dB
fast-1.25x 52.7 ms 0.992 −4.99 81.6 dB
fast-1.60x 68.4 ms 0.992 −4.84 81.2 dB
slow-0.80x 63.5 ms 0.990 −10.20 81.5 dB
music-under-20db 52.9 ms 0.991 0.00 31.2 dB
music-under-10db 52.9 ms 0.991 0.00 21.5 dB
music-under-5db 52.9 ms 0.991 0.00 16.8 dB
overlap-0.5s 513.8 ms 1.016 0.00 81.5 dB
overlap-2.0s 2014.4 ms 1.406 −14.02 81.5 dB

Hole 1 — time stretch produces false mispronunciation findings#

The fast and slow clips contain identical words from an identical speaker. Only the tempo changed. Yet 182.C.18.03's goodness-of-pronunciation falls to −4.99 at 1.25x and −10.20 at 0.80x, past its −3.0 floor, so the gate would report a mispronunciation that did not happen. A phase vocoder's transient smearing looks like bad pronunciation to a frame-level acoustic model.

assertGopMeasurable now refuses a pronunciation score on time-stretched audio. Refuses, not discounts: a discounted wrong number is still a wrong number.

The aligner, by contrast, held — 68.4 ms at 1.6x is well inside the ITU +45/−125 ms reject boundary — so timing claims are still allowed on it.

Hole 2 — overlapping speech breaks the aligner silently#

A second voice starting 0.5 s in gives a 513.8 ms onset error, four times the ITU reject boundary, and the aligner reports success. At 2.0 s it is 2014.4 ms with a 1.406 stretch slope. Forced alignment is given the words and asked where they are; it will always find them somewhere.

Worse, at the 0.5 s offset the GOP is exactly 0.00 — the take looks perfectly clean while its timing is half a second wrong. A clean-looking evaluator beside a broken one is how a bad take gets through.

assertAlignmentMeasurable and assertGopMeasurable both refuse on overlapping speech.

Not a hole, but worth knowing — music trips the SNR gate#

The aligner and the GOP are completely unaffected by a harmonic bed even 5 dB under the speech. But the percentile-split SNR estimator reads 16.8 dB there and the 182.C.18.04 low-snr gate fires at 20. Music under speech is not noise, and that estimator cannot tell the difference. The finding is not suppressed; snrFindingCaveat attaches the reason it may be the estimator rather than the take.

Reproducing#

text
python tools/human-video/evaluator/make_delivery_fixtures.py --out /tmp/delivery
python tools/human-video/evaluator/ctc_align.py --audio /tmp/delivery/overlap-0.5s.wav \
  --transcript "engine ships tuesday costs forty dollars"

The fixtures are seeded and not committed; the generator reproduces them.

Per-item status this evidence supports#

Item Was Is
182.C.18.08 not-started blocked-corpus: 4 of 10 conditions measured with real fixtures, 6 need 182.C.33 recorded speech
182.C.18.03 done now guarded — GOP refuses on time-stretched audio, after measuring false findings at every tempo factor
182.C.18.02 done now guarded — word timing refuses on overlapping speech, after a 513.8 ms silent error at a 0.5 s offset