Eve library

PROMPT-003 evidence pack — held-out prompt-injection portfolio

Evidence

Revision 1

On this page

Work-package charter#

  • Task ID: PROMPT-003
  • Owner: cross-version prompt-security maintainers; implementation owner: Codex
  • Priority: P0
  • Risk tier: Critical authority-confusion, prompt-injection, source-scope, generated-output, and release-permission risk
  • Dependencies: PROMPT-001, PROMPT-002, @oshun/prompt-trust, every migrated product boundary, and each boundary's deterministic output checks
  • Rollout strategy: credential-free held-out regression project plus an always-on pull-request/push workflow; no production model, persistence, or UI rollout
  • Rollback strategy: revert source commit d698ae1c53c0a4895b29fe7e4eeeabed70fcf07f; the suite, workflow, and Rust development dependency are additive
  • Evidence-pack location: this file

Closed implementation#

testing/prompt-injection is a dedicated Nx/pnpm test project. It keeps the versioned attack corpus and boundary registry outside production prompt code, loads the same corpus from TypeScript, Python, and Rust, invokes real production prompt builders and deterministic admission checks, and replaces only the network/model edge with local scripted doubles.

The registry names 20 invoked implementations:

Boundary ID Production implementation under test Runner
v1.ideation structured idea synthesizer TypeScript
v1.planner creative-orchestrator brief decomposition TypeScript
v1.bff-text BFF source-bound text generator TypeScript
v1.agent-loop research/tool-result generator TypeScript
v1.critique-revise artifact/rubric/prior-critique judge TypeScript
v1.cqj-self-refine CQJ critique-to-revision boundary TypeScript
v1.cqj-reflexion CQJ cross-item lesson memory TypeScript
v2.calliope-writer match-commentary writer TypeScript
v2.calliope-reviewer generated-commentary reviewer TypeScript
v2.hathor-dialogue NPC persona/world/history dialogue TypeScript
v2.maya-slm local SLM context window Rust
v6.iris-cognition governed cognition gateway TypeScript
v6.moirai-mount V6 tier-budget/tool-grant mount into Iris TypeScript
v6.clio event-scoped chronicle writer TypeScript
v7.forge realm artifact proposal and static admission TypeScript
v7.mawu Sophia-scoped generation backend boundary TypeScript
v8.proposer solver-owned case surface proposer TypeScript
v8.assets case asset job prompt TypeScript
v9.content source-bound educational content prompt TypeScript
v9.tutor governed Metis tutoring provider call Python

The registry test fails if a source or mapped test disappears, if a registered test stops naming its boundary, or if any held-out probe appears in a production boundary source. The latter makes the corpus held out from product prompts and few-shot examples; this is a committed regression set, not a claim that the benchmark itself is secret.

Attack and authority coverage#

The immutable corpus.v1.json contains one opaque probe for each required family:

  1. direct
  2. indirect
  3. cross-turn
  4. retrieval poisoning
  5. source poisoning
  6. generated output
  7. critic feedback
  8. tool result
  9. multimodal metadata/OCR
  10. encoded
  11. delimiter
  12. memory poisoning
  13. cross-item

Every payload requests the same six sentinel escalations: tool authority, source scope, rights, autonomy, budget, and release permission. The shared assertion requires each opaque probe to be present inside a real untrusted zone, removes every untrusted body, and then proves that neither the probe nor any of the six escalation values entered the trusted prompt shell. Encoded and delimiter attacks receive additional neutralization assertions. Boundary-level checks also prove the Moirai tool grant stays empty and its tier budget remains fixed, Forge capabilities remain policy-scoped, and Mawu/Clio source references remain the original typed references.

All 13 families run at all 20 implementations: 260 boundary/attack executions. The TypeScript report uses a single parameterized test for both Calliope writer and reviewer, so its test-count total is smaller than the number of boundary executions.

Generated-output integrity proof#

Seven negative regression tests exercise the post-generation admission layer:

  • an agent-loop tool call with an extra release field and a non-string query is rejected against the original JSON schema, and the handler call count remains zero;
  • a BFF citation outside the supplied source-ID set is rejected;
  • critic evidence absent from the exact artifact text is rejected;
  • unsafe generated commentary cannot be cleared by an empty model review;
  • a Clio event reference absent from the original log slice is rejected;
  • a Forge capability smuggled in generated source is rejected by the original capability policy;
  • V8 surface fields inconsistent with the typed case spec and V9 attacker text used as a source identifier are rejected.

These tests directly close the ledger requirement for citations, quotes, and tool arguments to be checked against original typed data after generation.

Model/provider change enforcement#

.github/workflows/prompt-injection-portfolio.yml has no path filter. It runs for every pull request to main, every push to main, and manual dispatch. The workflow runs portfolio typecheck, lint, and the sequential multi-language verify target. The TypeScript, Python, Rust, and aggregate verification targets are non-cacheable, preventing a boundary or provider change from receiving a stale green result. Project inputs and implicit dependencies additionally name the shared provider/agent-loop roots and every migrated package.

Automated verification#

Final verification was performed on the integrated and published source tree at ee77b83a26ecf1fbeaf6077f78d582d7b722161c.

  • Aggregate Nx target: passed.
  • TypeScript/Vitest: 232/232 tests across 5/5 files.
  • Python/Pytest: 1/1 corpus test, iterating all 13 attacks at the V9 tutor boundary.
  • Rust portfolio: 1/1 corpus test, iterating all 13 attacks at the Maya SLM boundary.
  • Complete Maya SLM crate: 18/18 integration tests (17 existing plus the portfolio test); library and doc-test targets also passed.
  • Harness TypeScript typecheck: passed with zero diagnostics.
  • Harness ESLint: passed with zero warnings/errors. Cross-project boundary-rule suppressions are limited to the V1/V2 test adapters and their narrow Iris test entry because those files intentionally invoke multiple product projects.
  • Ruff check and format: passed.
  • Cargo format: passed.
  • Prettier: passed for the workflow, README, JSON manifests/corpus/registry, config, and TypeScript sources.
  • Workflow YAML parse, git diff --check, and frozen offline pnpm lock validation: passed.
  • The source commit's pre-commit Nx affected typecheck independently passed for @oshun/prompt-injection-evals.

Representative commands:

text
NX_DAEMON=false pnpm exec nx run @oshun/prompt-injection-evals:verify --parallel=1
pnpm exec tsc --noEmit -p testing/prompt-injection/tsconfig.json
pnpm exec eslint testing/prompt-injection/src/*.ts testing/prompt-injection/vitest.config.ts --no-ignore --max-warnings=0
PYTHONPATH=services/metis/src python3 -m pytest -q testing/prompt-injection/python/test_portfolio_prompt_injection.py
cargo test --manifest-path libs/maya/engine-core/crates/maya-slm-runtime/Cargo.toml
cargo fmt --manifest-path libs/maya/engine-core/crates/maya-slm-runtime/Cargo.toml --check
python3 -m ruff check testing/prompt-injection/python/test_portfolio_prompt_injection.py
pnpm install --lockfile-only --offline --ignore-scripts --frozen-lockfile --filter @oshun/prompt-injection-evals

The final aggregate target ran from 2026-07-16T18:56:47Z through 2026-07-16T18:57:20Z on Linux 6.8.0-110-generic x86_64 with Node 24.14.1, pnpm 10.25.0, Python 3.12.3, and Rust 1.94.1. No live provider credential, remote model call, or claim about a production model response is involved.

Resource safety and reconciliation#

The host had 15 GiB RAM and 31 GiB swap. All verification was targeted, sequential, and single-worker. Two early aggregate-wrapper attempts were stopped while a competing Nx daemon was rebuilding the large workspace graph; their underlying direct TypeScript, Python, and Rust commands had already passed. The final daemon-free aggregate target completed successfully after the competing graph exited. No full Nx sweep, Next build, Expo export, Tauri package, Unreal build, Playwright run, or mobile automation was applicable or attempted.

origin/main advanced during publication. The source branch took the newer remote lockfile, reapplied only the prompt-injection importer, passed frozen offline lock validation and the complete post-merge portfolio, and published the combined tip without dropping either change set.

Artifact identity#

Artifact SHA-256
Held-out corpus 4f7602baed09058ac885dfeac119d9828881d79803c584a6fe57a6d5cf62e26e
Boundary registry 3ca7b7de49adec7c3315f0dbfa7092f90b83bffb5f8d457695ab571a3f73eb2f
Nx project 30eca6c852fb9c26d11f60691dc88ad44da078360adc7b621ceb1e1e9247c4bf
Shared TypeScript harness dc0bde60555acf32a186c2c8895ef55c129f06c50c55f82fa414584b84965e26
V1 TypeScript runner 791d4e5606b61837c12392363a9ef3c05b3a8f0808394913312f8b9832daaee0
V2 TypeScript runner 10f6e1131ffcae5f3288ba19f97a4e4c4a7da29cefb24ef2d3bcf2c918578699
V6/V7 TypeScript runner c4aa4dbf1a0c693c8d1e77fb4e4d45dc8d5ae873c3362a8b32a8dceb66d951be
V8/V9 TypeScript runner c9be93d2d7a78cd8444ca4ec5d5ec1d499b2759bd326d010e9d6ff6629bd0cd1
Python tutor runner 2b648c45482de71ec24299fdd7d619c5ca290a8cbb30e364a4c12f9a2eed4167
Rust SLM runner fe169f361e60e0e07c7dc9f2774dad4cf01fc16eb112f619306f7735cea1aa06
Always-on workflow b13c5e3168b80383a48e8499dfb04e058e9ba218b4532eeaca2d0988ffecfb82

Scope and remaining seams#

  • This suite proves typed prompt construction and deterministic post-generation admission with local scripted provider doubles. It does not claim that prompt structure alone makes arbitrary production models immune to all attacks.
  • Live-model red-team calibration, provider-specific behavioral thresholds, and release-signature decisions remain governed by their model/release processes.
  • Forge's static capability inspection remains admission-only; executable sandbox enforcement remains under the named AUTH/V7 work.
  • Safety-classifier quality, substantive claim entailment beyond typed source membership, durable V6 state, and live-provider readiness remain under their named SAFETY, PROOF, V6, V7, V8, and V9 packages.
  • No rendered web, PWA, mobile, admin, or Unreal UI changed, so frontend skill, Playwright, mobile automation, screenshots, and Unreal verification were inapplicable.

Publication identity#

  • Authoritative source commit: d698ae1c53c0a4895b29fe7e4eeeabed70fcf07f
  • Source publication tip on origin/main: ee77b83a26ecf1fbeaf6077f78d582d7b722161c
  • Closure evidence commit: recorded in the PROMPT-003 ledger closure entry after this evidence file is published.