Task 3.1 independent relevance-label review packet

Status: awaiting two independent human reviews. This packet contains no proposed relevance labels and is not eligible human evidence.

Set: eve.relevance-set.v1
Set source SHA-256: 9863f64a266171ba76c5276c631d0c874496bc1aaf1f49d389de9320ccba295e
Inventory SHA-256: 7da1f77c35b847787f29ea34f0f1784e047421c5aee39c345a1dd87dbe65ccf5
Inventory content hash: 3d5f56907f27 (3256 pages)
Member corpus: 0 pages / 0 chunks

Protocol#

Assign this packet to two distinct human reviewers. They must work independently.
Inspect the source documents directly. Do not use the BM25F system under evaluation, its result order, or model-generated relevance proposals.
For every query, record every relevant document as its exact href from the bound inventory. Lists must be non-empty, unique, and lexicographically sorted.
Each reviewer records their own rationale. The two href lists must match exactly; any mismatch requires independent human adjudication.
Create human-labels.reviewed.json beside this packet with schema eve-sota-relevance-human-labels.v1, the exact corpusBinding from the pending JSON, a canonical reviewedAt, and two reviewer objects containing the fields and labels shown by the pending template.
Run node tools/eve-everywhere/verify-relevance-set-human-review.mjs. A passing receipt can then be admitted into the versioned relevance set; it does not by itself close downstream retrieval-quality tasks.

Queries#

concept-admin-intent-plane#

Class: conceptual
Audience: admin
Query: According to the documentation, how does the builder workbench intent plane work?

Reviewer 1 hrefs and rationale:

Reviewer 2 hrefs and rationale:

concept-admin-reading-positions#

Class: conceptual
Audience: admin
Query: how do reading positions work?

Reviewer 1 hrefs and rationale:

Reviewer 2 hrefs and rationale:

multihop-admin-lease-to-verify#

Class: multi-hop
Audience: admin
Query: What has to happen between an agent taking a lease and an item reaching verified?

Reviewer 1 hrefs and rationale:

Reviewer 2 hrefs and rationale:

recency-admin-current-gates#

Class: recency
Audience: admin
Query: Which ADR-0076 implementation gates are still open?

Reviewer 1 hrefs and rationale:

Reviewer 2 hrefs and rationale:

contradictory-admin-member-corpus#

Class: contradictory
Audience: admin
Query: Is the member documentation corpus the curated docs-center, or is it empty?

Reviewer 1 hrefs and rationale:

Reviewer 2 hrefs and rationale:

Required attestation per reviewer#

Identify yourself and your role, confirm that you manually inspected source documents for all five queries, worked independently of the other reviewer, used neither the evaluated BM25F ranker nor model-generated labels, and stand behind every href and rationale in your label set.