# Eve SOTA retrieval relevance set — `eve.relevance-set.v1`

- **Task:** 3.1
- **Evaluated:** 2026-09-05
- **Decision:** relevance-set-versioned-labels-independent
- **System under evaluation:** `assistant-docs-search:bm25f`
- **Record digest:**
  `20926cf29e0fd9531818c55003d0010090758d22348141edd7607355199151d0`

## The rule

A versioned relevance set exists over the nine query classes task 3.1 names,
every query carries a provenance pointing at somewhere a real question was
asked, and no label came from the ranker the set judges. Five classes are
settled by oracles independent of ranking — corpus identity, corpus absence, and
the audience boundary — and are re-derived from the corpus itself whenever it is
present. The four classes that need a person are recorded as awaiting one and
carry no labels at all, because a machine guess wearing a human label is worse
than an admitted gap.

The class list, audience list, label-source vocabulary and every query are read
out of the set module; the corpus-derived labels are re-checked line by line
against the built corpus, and the record says plainly when that corpus is absent
rather than reporting a check nobody ran.

## The nine classes

| Class             | Queries | Labelled | Awaiting a human |
| ----------------- | ------- | -------- | ---------------- |
| navigational      | 2       | 2        | 0                |
| exact-id          | 2       | 2        | 0                |
| conceptual        | 2       | 0        | 2                |
| multi-hop         | 1       | 0        | 1                |
| recency           | 1       | 0        | 1                |
| contradictory     | 1       | 0        | 1                |
| empty-unknown     | 1       | 1        | 0                |
| tenant-sensitive  | 1       | 1        | 0                |
| injection-bearing | 1       | 1        | 0                |

7 of 12 queries are labelled from an oracle independent of ranking; 5 await a
person and carry no labels at all.

## Independence from the system under evaluation

- The ranker is not a declared label source: true
- No label cites it: true
- Independence is enforced in code, not promised: true
- A contaminated label is proved caught: true

## Honest states

- The four human-judgement classes await a human: true
- No query claims a human label: true
- Member retrieval is absent by construction, not broken: true
- "Cannot answer" is distinguished from "nobody judged": true

## Corpus binding

- Full corpus present at generation: true
- Chunks: 57008
- Docs inventory hash: `3d5f56907f27` over 3256 pages
- Member corpus: 0 pages / 0 chunks

| Re-derived label                                 | Subject                                                  | Chunks matching | Holds | Self-referential chunks ignored                       |
| ------------------------------------------------ | -------------------------------------------------------- | --------------- | ----- | ----------------------------------------------------- |
| corpus-identifier (nav-admin-repo-map)           | `docs-center/reference/repo-map.html`                    | 25              | true  | 0                                                     |
| corpus-identifier (id-admin-adr-0076)            | `docs/adr/ADR-0076-eve-governed-delivery-lifecycle.html` | 24              | true  | 0                                                     |
| corpus-absence (empty-admin-kubernetes-operator) | `kubernetes operator`                                    | 0               | true  | 1 (`docs/audits/EVE_SOTA_RELEVANCE_SET_2026-09.html`) |

## Honest limits

- This record closes task 3.1 only. It prices no embedding model (3.2), builds
  no dense index (3.3), fuses nothing (3.4), measures no Recall@k or nDCG (3.5),
  runs no poisoning or adversarial suite (3.6), promotes nothing (3.7), and adds
  no rebuild or drift operations (3.8).
- Four of the nine classes — conceptual, multi-hop, recency and contradictory —
  have NO relevance labels. They need a person, nobody in this repository can be
  that person, and the set records the gap rather than filling it. Any
  measurement over those classes today would be measuring nothing.
- Two labelled queries carry a deliberately empty relevant list with the reason
  written out, because the oracle that could settle them exists while the
  selection does not. They are not scored as "nothing is relevant"; the
  distinction is in the rationale and is the difference between an absence and
  an unmade judgement.
- The member audience has no retrieval at all — the member corpus is empty by
  construction and search_docs is withheld from member sessions — so its queries
  record a policy outcome rather than a ranking one. If a page ever opts into
  the member audience, those labels must be revisited.
- The corpus-derived labels are re-checked by scanning the built corpus for an
  exact href and an exact term. That settles identity and absence; it does not
  settle whether a document that mentions a term is a GOOD answer, which is
  precisely what the human classes are for.
- The full corpus is gitignored at roughly 46 MB, so a checkout without it
  cannot re-derive the corpus labels. The record states when the check was
  skipped instead of implying it passed.
- The absence oracle excludes pages under docs/audits/ that name the absence
  query's own id: once this record is rendered into the estate it labels, the
  corpus contains a page that prints the query, and an oracle that counted it
  would refute itself on every rebuild. The excluded pages and chunk counts are
  on the finding.