Eve library

Metis Publication Release Gates Runbook

Guide

Revision 1

On this page

Status: Canonical
Last updated: 2026-04-11
Owners: Metis Engineering, Metis Operations, Metis Safety, Metis Rights Review

Purpose#

This runbook defines how Metis operators, instructors, and release owners evaluate and resolve publication blockers before learner-facing courses are allowed to ship.

It is aligned to the enforced publication gates in:

  • services/metis/src/metis/services/publication_release_gates.py
  • services/metis/src/metis/services/course_service.py
  • services/metis/src/metis/services/admin_service.py

Gate Model#

Metis course publication is fail-closed. A course may not move to published unless every release gate below is green:

  1. groundedness The learner-facing lesson set must meet the grounded lesson bar. Current enforced threshold: at least 80% grounded lesson coverage and at least one published lesson.
  2. provenance Source coverage must be strong enough for operator inspection and later export lineage. Current enforced threshold: at least 80% source coverage.
  3. safety The course must be in a review-ready posture with real learner-facing content, not an empty or bypassed draft.
  4. rights Source rights must be cleared with zero unknown-rights carryover. Publication fails if rights are blocked or any source remains unknown.
  5. performance Release readiness must not carry degraded runtime signals at publish time. Current enforced threshold: all modules published, no known escalated tutoring spillover in the admin approval path.
  6. automation_coverage Learner-facing assessment coverage must be present before publication. Current enforced threshold: at least one assessment, at least one published assessment, and at least 80% assessment support.

Where Operators Work#

  • Instructor publish path: PATCH /api/v1/courses/{course_id} with status=published
  • Admin approval path: POST /api/admin/courses/{course_id}/approval
  • Review queues: /reviews
  • Source rights workspace: /rights
  • Incident command: /incidents
  • Monitoring and release-gate stream: /monitoring
  • Operations lane: /operations

Failure Handling#

When publication is blocked, Metis returns HTTP 409 Conflict and an operator- readable release-gate summary.

Default triage order:

  1. Confirm the blocking gate named in the response.
  2. Open the matching workspace.
  3. Resolve the missing evidence, rights posture, review state, or assessment coverage.
  4. Retry publication only after the underlying signal is actually green.

Do not override a blocked course by writing directly to the database or by manual status edits outside the governed admin path.

Gate-by-Gate Triage#

Groundedness#

Symptoms:

  • grounded lesson coverage below 80%
  • publication package queue shows low-confidence or abstention-required output

Actions:

  1. Open /reviews?queue=grounded_build.
  2. Inspect evidence bundles for published lessons and assessments.
  3. Unpublish or revise unsupported lessons.
  4. Re-run grounded build review until confidence and evidence posture recover.

Provenance#

Symptoms:

  • source coverage below 80%
  • learner-facing materials lack inspectable source backing

Actions:

  1. Open /rights.
  2. Confirm every learner-facing lesson has source-backed content or attached references.
  3. Fill missing source links and provenance metadata.
  4. Recheck the review queue after coverage improves.

Safety#

Symptoms:

  • course is still effectively a draft
  • review-ready structure is incomplete

Actions:

  1. Confirm at least one real learner-facing lesson exists.
  2. Ensure the course has entered a reviewable posture instead of remaining an empty shell.
  3. Route unresolved quality issues through /reviews.

Rights#

Symptoms:

  • rights posture is blocked
  • one or more sources still show unknown rights

Actions:

  1. Open /rights.
  2. Resolve restricted, unknown, or missing source-license signals.
  3. Capture the allowed-use rationale in the source-rights workspace.
  4. Retry publication only after rights posture is cleared and unknown rights count is zero.

Performance#

Symptoms:

  • modules are not fully publish-ready
  • admin approval path shows escalated tutoring spillover tied to the course

Actions:

  1. Open /operations and /monitoring.
  2. Confirm the issue is not active learner-visible degradation.
  3. Clear module publish gaps or route the runtime issue into incident command.
  4. Return only when the course no longer contributes to blocked release posture.

Automation Coverage#

Symptoms:

  • no assessment exists
  • no assessment is published
  • assessment support remains below 80%

Actions:

  1. Open the course assessment surface.
  2. Publish or complete the required learner-facing assessments.
  3. Confirm the publication package queue reflects assessment support at or above the threshold.

Incident Escalation Rules#

Escalate immediately when any of the following are true:

  • rights posture is blocked for a course already promoted externally
  • groundedness blockers affect a live learner entry point
  • policy-sensitive tutoring spillover appears during release review
  • the same course fails publication twice in one shift without net signal improvement
  • operators suspect manual status bypass or stale approval evidence

Primary incident categories:

  • grounding_failure
  • build_failure
  • export_failure
  • source_rights
  • tutoring_latency

Evidence Required For A Clean Publish#

  • source-backed lesson coverage at or above threshold
  • grounded lesson coverage at or above threshold
  • published assessment support at or above threshold
  • cleared source-rights posture with zero unknown-rights carryover
  • no unresolved admin release blocker for the course
  • clear audit trail showing who approved the transition

Audit Expectations#

Every approval or blocked publication must leave:

  • the actor
  • the action
  • the target course
  • the reason or reviewer comment
  • the workspace path used for remediation

Verification#

This runbook is aligned to the enforced backend behavior validated by:

  • pytest services/metis/tests/test_admin_review_gates.py services/metis/tests/test_backend_service_coverage.py
  • ruff check services/metis/src/metis/services/publication_release_gates.py services/metis/src/metis/services/course_service.py services/metis/src/metis/api/courses.py services/metis/src/metis/services/admin_service.py services/metis/src/metis/api/admin.py services/metis/tests/test_backend_service_coverage.py services/metis/tests/test_admin_review_gates.py