Eve library

CPU sync-evaluator and forced-alignment probe — 2026-09-10

Guide

Revision 1

On this page

Evidence for 182.C.17 (sync evaluators), 182.C.18.02 (word alignment), and the phase-level hardware constraint. Third correction of the day, same cause: all thirteen 182.C.17 rows, all eight 182.C.20 rows, and all eight 182.C.32 rows carried one copied blocker, "No GPU host; needs a RunPod worker per 182.C.17.13", and 182.C.18.02 was blocked on "a forced aligner on the hosted GPU worker". None of those items had been tried. Both premises were tested here and both are false for the evaluator side.

Machine#

Linux, 15 GiB RAM, 4 cores, no GPU (nvidia-smi absent; torch 2.14.0+cpu, faster-whisper 1.2.1, transformers 5.17.0, onnxruntime 1.25.1 installed system-wide). free -m before the runs: 10.9 GB available.

1. SyncNet v2 runs on this CPU#

joonson/syncnet_python — the LSE-C / LSE-D baseline the phase names in 182.C.17.01. MIT licence (LICENSE.md, copyright Joon Son Chung). Weights syncnet_v2.model, 54,573,114 bytes, sha256 961e8696f888fce4f3f3a6c3d5b3267cf5b343100b238e79b2659bff2c605442, fetched from https://www.robots.ox.ac.uk/~vgg/software/lipsync/data/syncnet_v2.model. Loads with strict=True: 13.63 M parameters.

Forward pass on CPU, 4 threads, torch.no_grad(), batch of 20 windows:

Stream Input shape Output Time
lip (forward_lip) [20, 3, 5, 224, 224] [20, 1024] 0.80 s
audio (forward_aud) [20, 1, 13, 20] [20, 1024] 0.31 s
per second of 25 fps video (25 five-frame windows) — — ≈ 1.0 s CPU (lip) + 0.4 s (audio)

A 10 s candidate therefore costs about 15 s of CPU to score end to end, before face tracking. That is a throughput number for 182.C.17.12, not a blocker. The hosted GPU worker in 182.C.17.13 is a production fan-out decision to be taken from measured throughput, not a prerequisite for building the evaluator.

Not tested here: a real talking-face clip through the full pipeline (face crop → MFCC → SyncNet → offset/confidence). No generated human video exists in the repo yet, and producing one is provider spend under prohibition 2 (consent binding before any reference upload). The forward pass on correctly shaped tensors proves the compute premise only.

2. A CTC forced aligner runs on this CPU, and it is accurate#

facebook/wav2vec2-base-960h through transformers (Apache-2.0; the masked_spec_embed weight is reported missing at load, which only affects training-time masking), with the standard CTC trellis and backtracking. 20.1 ms frames.

Ground truth#

Timestamps from a chat model are generated, and Whisper's are inferred from cross-attention; to measure either you need onsets known to the sample. So: Kokoro-82M (hexgrad/Kokoro-82M, cached locally, voice af_heart) synthesized six words separately, each trimmed to its first and last sample above |0.01|, then concatenated with exactly 600 ms of digital silence between them. Every onset is therefore known exactly (synth-truth.json; known-onset-clip-16k.wav, 7.23 s).

Results (align-accuracy.json)#

Aligner CPU time Onset error per word (ms) Max abs onset Word-end error (ms)
wav2vec2 CTC forced alignment 1.29 s for 7.23 s +81, +65, +17, +17, +62, +35 81 ms −96 … −141 (consistently early)
faster-whisper tiny.en, word_timestamps 2.0 s (3.95 s clip) −600, +102, +532, +151, +113 600 ms merged "forty dollars" → "$40"
faster-whisper base.en 5.7 s −600, −58, −128, −49, −87, −116 600 ms −218 … −352
faster-whisper small.en 15.3 s −600, +182, +52, +171, +33 600 ms merged "forty dollars" → "$40"

Against the thresholds in 182.C.03.04 (ATSC IS-191 target +15/−45 ms; ITU-R BT.1359-1 reject +45/−125 ms):

  • CTC is inside the ITU reject boundary on every word, raw. Its error is a systematic late bias (every onset positive, 17–81 ms), so a calibrated constant corrects most of it. That calibration was run — it is the first step of 182.C.18.02 — and it corrects an earlier claim on this page, which said the constant brings CTC inside the ATSC window. It does not. aligner-calibration.json (tools/human-video/evaluator/calibrate_aligner.py, reproducing the onsets above exactly): median bias +48.288 ms, so the correction is −48.288 ms; the residual scatter that remains is ±32.87 ms (worst lead −31.68, worst lag +32.87). ATSC IS-191 allows only 15 ms of lead, and the corrected worst lead is 31.68 — outside ATSC. It is comfortably inside ITU-R BT.1359-1 (+45/−125) with ~30% margin on the tighter side. That ±33 ms is the aligner's own noise floor and therefore the floor on any timing claim built from it. Word ends are consistently ~100–140 ms early (CTC emits a token before the sound finishes), so use onsets, not ends, for offset work.
  • Whisper's timestamps are disqualified, like the OpenRouter chat models' in ../2026-09-10-openrouter-asr/: the first word is snapped to 0.0 (600 ms early, every model size), mid-clip onsets are off by up to 532 ms, and two of three sizes merged two spoken words into one orthographic token, which also breaks word counting. Whisper remains fine for transcription; it is not an aligner.
  • On the real probe.wav (24 kHz TTS, ../2026-09-10-openrouter-asr/), CTC aligned the spoken words forty two dollars as three words in 0.68 s (fw-words.json has the Whisper timings for the same clip); a chat model returns $42 and the numbers-to-words expansion in dialogue.ts must run before alignment.

Pronunciation, from the same posteriors (pronunciation-gop.json)#

182.C.18.03 asks a different question from 182.C.18.01: not whether the recognizer wrote the right letters, but whether the ACOUSTICS support them. A recognizer's language model repairs a mangled brand name into the spelling it expects, so a clean transcript is not evidence the name was said right.

The measure is Witt & Young goodness of pronunciation over the same CTC model, GOP = log P(canonical | emission frame) - max_s log P(s | emission frame). At most 0; 0 means the acoustic model would itself have chosen that character.

Clip Correct words One word substituted, same audio
known-onset-clip-16k.wav all six at exactly 0.0 tuesday→thursday −15.32, costs→coasts −9.62, forty→fourteen −11.17, engine→enjoying −20.18
../2026-09-10-openrouter-asr/probe.wav (24k→16k) all ten at exactly 0.0 —

In every substitution the other five words stayed at exactly 0.0, so the fault localises to a word and to a character within it (thursday fails at R, where the model preferred E).

The finding, and it constrains how this can be used. The separation is total, so any floor in the gap separates these cases and the measurement does not say which value is right. More importantly the measure SATURATES: two different voices and two different sentences all sit at exactly 0.0, so there is no headroom above zero in which to rank a perfect pronunciation against an accented but acceptable one. It is a detector, not a grade. Both fixtures are clean synthetic English, so they do not establish a threshold for real human speech at all — that needs the 182.C.33 rated corpus, the same blocker as 182.C.17.08. The shipped floor is −3.0 and is labelled provisional in code, placed nearer the correct end deliberately, because a false "mispronounced" on a real accent rejects a take a human would accept.

Locale is the other half. wav2vec2-base-960h is LibriSpeech English. Given a Yoruba name it has no symbols for those sounds, and a low GOP then means the model does not know the language rather than that the speaker said it wrongly. Those terms are reported UNMEASURABLE with no number attached and routed to 182.C.18.07; unmeasurable and failed must never look alike.

A defect this work found in the aligner committed hours earlier. tokenize silently dropped any character outside the model vocabulary. Aligning "costs 40 dollars" against audio saying "costs forty dollars" dropped both digits, put 40 463 ms late, and lost dollars from the output entirely. It now refuses with unalignable_character, and merge_words groups by the labels the tokenizer produced instead of counting transcript characters in parallel — the two walks agreeing was the assumption that broke. The 182.C.18.02 calibration onsets are byte-identical after the rewrite.

Per-item status this evidence supports#

Item Was Is
182.C.17.01–.12 blocked-local, "No GPU host" not-started; SyncNet v2 measured on this CPU; second evaluator still to be selected and licensed
182.C.17.13 blocked-local, "Mac cannot host SyncNet" not-started; a throughput/hosting decision taken from 182.C.17.12 numbers
182.C.18.02 blocked-local, aligner needs GPU worker done; CTC aligner on this CPU, calibrated −48.288 ms, ±32.87 ms residual (inside ITU, outside ATSC)
182.C.18.03 not covered done; GOP from the same posteriors, saturated on clean speech, threshold provisional pending 182.C.33
182.C.20.01–.08 blocked-local, "No GPU host" not-started; depends on 182.C.17 and on real generated active-scene clips (spend, consent-bound)
182.C.32.01–.02 blocked-local, "GPU-bound research" not-started; a benchmark-harness definition and a licence review need no GPU
182.C.32.03–.08 blocked-local, "GPU-bound research" blocked-local, now justified: self-hosted diffusion video generators need CUDA; none here

Where the weights are#

Staged outside every worktree, so a scratchpad clean-up cannot delete them: ~/.cache/oshun-human-video/ — yunet.onnx (8f2383e4…), sface.onnx (0ba9fbfa…), face_landmarker.task (64184e22…), syncnet_v2.model (961e8696…), SyncNetModel.py (41b85aed…), LICENSE.md. 93 MB total. Nothing under that directory is committed; 182.C.17.02 re-fetches by digest.

The evaluator interpreter is ~/.cache/oshun-human-video/venv — CPU torch, transformers, scipy, python_speech_features. It is the single interpreter every evaluator script runs under, including the aligner, and the specs that shell out to those scripts skip visibly when it is absent rather than passing quietly.

Reproducing#

The alignment and calibration scripts are now committed and rerun the numbers above from the committed fixture:

text
tools/human-video/evaluator/ctc_align.py        --audio known-onset-clip-16k.wav --transcript "engine ships tuesday costs forty dollars"
tools/human-video/evaluator/calibrate_aligner.py --audio known-onset-clip-16k.wav --truth synth-truth.json --out aligner-calibration.json

The rest lived in the session scratchpad and is reproduced from the numbers above: SyncNet forward pass with random tensors of the shapes listed; Kokoro per-word synthesis with 600 ms gaps; faster-whisper with word_timestamps=True, beam_size=5, compute_type=int8. No provider spend.