Status: Canonical
Last updated: 2026-04-11
Owners: Metis Engineering, Metis Operations, Metis Safety, Metis Rights Review
Purpose#
This runbook defines how Metis operators, instructors, and release owners evaluate and resolve publication blockers before learner-facing courses are allowed to ship.
It is aligned to the enforced publication gates in:
services/metis/src/metis/services/publication_release_gates.pyservices/metis/src/metis/services/course_service.pyservices/metis/src/metis/services/admin_service.py
Gate Model#
Metis course publication is fail-closed. A course may not move to published
unless every release gate below is green:
groundednessThe learner-facing lesson set must meet the grounded lesson bar. Current enforced threshold: at least 80% grounded lesson coverage and at least one published lesson.provenanceSource coverage must be strong enough for operator inspection and later export lineage. Current enforced threshold: at least 80% source coverage.safetyThe course must be in a review-ready posture with real learner-facing content, not an empty or bypassed draft.rightsSource rights must be cleared with zero unknown-rights carryover. Publication fails if rights are blocked or any source remains unknown.performanceRelease readiness must not carry degraded runtime signals at publish time. Current enforced threshold: all modules published, no known escalated tutoring spillover in the admin approval path.automation_coverageLearner-facing assessment coverage must be present before publication. Current enforced threshold: at least one assessment, at least one published assessment, and at least 80% assessment support.
Where Operators Work#
- Instructor publish path:
PATCH /api/v1/courses/{course_id}withstatus=published - Admin approval path:
POST /api/admin/courses/{course_id}/approval - Review queues:
/reviews - Source rights workspace:
/rights - Incident command:
/incidents - Monitoring and release-gate stream:
/monitoring - Operations lane:
/operations
Failure Handling#
When publication is blocked, Metis returns HTTP 409 Conflict and an operator-
readable release-gate summary.
Default triage order:
- Confirm the blocking gate named in the response.
- Open the matching workspace.
- Resolve the missing evidence, rights posture, review state, or assessment coverage.
- Retry publication only after the underlying signal is actually green.
Do not override a blocked course by writing directly to the database or by manual status edits outside the governed admin path.
Gate-by-Gate Triage#
Groundedness#
Symptoms:
- grounded lesson coverage below 80%
- publication package queue shows low-confidence or abstention-required output
Actions:
- Open
/reviews?queue=grounded_build. - Inspect evidence bundles for published lessons and assessments.
- Unpublish or revise unsupported lessons.
- Re-run grounded build review until confidence and evidence posture recover.
Provenance#
Symptoms:
- source coverage below 80%
- learner-facing materials lack inspectable source backing
Actions:
- Open
/rights. - Confirm every learner-facing lesson has source-backed content or attached references.
- Fill missing source links and provenance metadata.
- Recheck the review queue after coverage improves.
Safety#
Symptoms:
- course is still effectively a draft
- review-ready structure is incomplete
Actions:
- Confirm at least one real learner-facing lesson exists.
- Ensure the course has entered a reviewable posture instead of remaining an empty shell.
- Route unresolved quality issues through
/reviews.
Rights#
Symptoms:
- rights posture is
blocked - one or more sources still show unknown rights
Actions:
- Open
/rights. - Resolve restricted, unknown, or missing source-license signals.
- Capture the allowed-use rationale in the source-rights workspace.
- Retry publication only after rights posture is
clearedand unknown rights count is zero.
Performance#
Symptoms:
- modules are not fully publish-ready
- admin approval path shows escalated tutoring spillover tied to the course
Actions:
- Open
/operationsand/monitoring. - Confirm the issue is not active learner-visible degradation.
- Clear module publish gaps or route the runtime issue into incident command.
- Return only when the course no longer contributes to blocked release posture.
Automation Coverage#
Symptoms:
- no assessment exists
- no assessment is published
- assessment support remains below 80%
Actions:
- Open the course assessment surface.
- Publish or complete the required learner-facing assessments.
- Confirm the publication package queue reflects assessment support at or above the threshold.
Incident Escalation Rules#
Escalate immediately when any of the following are true:
- rights posture is blocked for a course already promoted externally
- groundedness blockers affect a live learner entry point
- policy-sensitive tutoring spillover appears during release review
- the same course fails publication twice in one shift without net signal improvement
- operators suspect manual status bypass or stale approval evidence
Primary incident categories:
grounding_failurebuild_failureexport_failuresource_rightstutoring_latency
Evidence Required For A Clean Publish#
- source-backed lesson coverage at or above threshold
- grounded lesson coverage at or above threshold
- published assessment support at or above threshold
- cleared source-rights posture with zero unknown-rights carryover
- no unresolved admin release blocker for the course
- clear audit trail showing who approved the transition
Audit Expectations#
Every approval or blocked publication must leave:
- the actor
- the action
- the target course
- the reason or reviewer comment
- the workspace path used for remediation
Verification#
This runbook is aligned to the enforced backend behavior validated by:
pytest services/metis/tests/test_admin_review_gates.py services/metis/tests/test_backend_service_coverage.pyruff check services/metis/src/metis/services/publication_release_gates.py services/metis/src/metis/services/course_service.py services/metis/src/metis/api/courses.py services/metis/src/metis/services/admin_service.py services/metis/src/metis/api/admin.py services/metis/tests/test_backend_service_coverage.py services/metis/tests/test_admin_review_gates.py