Eve SOTA retrieval relevance set — eve.relevance-set.v1

• Task: 3.1
• Evaluated: 2026-09-05
• Decision: relevance-set-versioned-labels-independent
• System under evaluation: assistant-docs-search:bm25f
• Record digest: 20926cf29e0fd9531818c55003d0010090758d22348141edd7607355199151d0

The rule#

A versioned relevance set exists over the nine query classes task 3.1 names, every query carries a provenance pointing at somewhere a real question was asked, and no label came from the ranker the set judges. Five classes are settled by oracles independent of ranking — corpus identity, corpus absence, and the audience boundary — and are re-derived from the corpus itself whenever it is present. The four classes that need a person are recorded as awaiting one and carry no labels at all, because a machine guess wearing a human label is worse than an admitted gap.

The class list, audience list, label-source vocabulary and every query are read out of the set module; the corpus-derived labels are re-checked line by line against the built corpus, and the record says plainly when that corpus is absent rather than reporting a check nobody ran.

The nine classes#

7 of 12 queries are labelled from an oracle independent of ranking; 5 await a person and carry no labels at all.

Independence from the system under evaluation#

• The ranker is not a declared label source: true
• No label cites it: true
• Independence is enforced in code, not promised: true
• A contaminated label is proved caught: true

Honest states#

• The four human-judgement classes await a human: true
• No query claims a human label: true
• Member retrieval is absent by construction, not broken: true
• "Cannot answer" is distinguished from "nobody judged": true

Corpus binding#

• Full corpus present at generation: true
• Chunks: 57008
• Docs inventory hash: 3d5f56907f27 over 3256 pages
• Member corpus: 0 pages / 0 chunks

Honest limits#

• This record closes task 3.1 only. It prices no embedding model (3.2), builds no dense index (3.3), fuses nothing (3.4), measures no Recall@k or nDCG (3.5), runs no poisoning or adversarial suite (3.6), promotes nothing (3.7), and adds no rebuild or drift operations (3.8).
• Four of the nine classes — conceptual, multi-hop, recency and contradictory — have NO relevance labels. They need a person, nobody in this repository can be that person, and the set records the gap rather than filling it. Any measurement over those classes today would be measuring nothing.
• Two labelled queries carry a deliberately empty relevant list with the reason written out, because the oracle that could settle them exists while the selection does not. They are not scored as "nothing is relevant"; the distinction is in the rationale and is the difference between an absence and an unmade judgement.
• The member audience has no retrieval at all — the member corpus is empty by construction and search_docs is withheld from member sessions — so its queries record a policy outcome rather than a ranking one. If a page ever opts into the member audience, those labels must be revisited.
• The corpus-derived labels are re-checked by scanning the built corpus for an exact href and an exact term. That settles identity and absence; it does not settle whether a document that mentions a term is a GOOD answer, which is precisely what the human classes are for.
• The full corpus is gitignored at roughly 46 MB, so a checkout without it cannot re-derive the corpus labels. The record states when the check was skipped instead of implying it passed.
• The absence oracle excludes pages under docs/audits/ that name the absence query's own id: once this record is rendered into the estate it labels, the corpus contains a page that prints the query, and an oracle that counted it would refute itself on every rebuild. The excluded pages and chunk counts are on the finding.