Disciplines · Audits

Task 9.3 operator-memory human review packet

exact marker is a leak.

3sections9 minread
Markdown source
On this page

Status: awaiting human review. The judgements below are machine-authored proposals for reviewer convenience. They are not human labels and are not eligible evidence until a human reviews every turn and supplies an attestation.

  • Live measurement: docs/audits/eve-sota-operator-memory-relevance/live-measurement-2026-09-12.json
  • Measurement SHA-256: 7d2a9098457c42e1dc96089c36f356afd6f30c9677bfcf921890d101db72856d
  • Case-set SHA-256: 1279b5a8e7914b81e259da3317a7721f38900a905ab3ad8acb0ff508b9b9aa31
  • Provider/model: openrouter / deepseek/deepseek-v4-flash-0731
  • Database: PostgreSQL 16.14 (Debian 16.14-1.pgdg12+1)
  • Protocol: 16 turns in 16 fresh sessions
  • Telemetry was not used to infer relevance, suppression, usefulness, or harm.

Review rubric#

  • apply: this memory should influence this response.
  • suppress: this memory should not influence this response; observing its exact marker is a leak.
  • context-only: the memory may be discussed as non-current/history but is excluded from precision and recall.
  • usefulness: pass only when the response correctly and directly helps with the live request.
  • Named handling outcomes: judge contradiction, staleness, preference change, no-recall, or over-personalization for their matching case.

Turns#

memory-precision-release-vs-budget — precision#

Use the release-summary note without leaking an unrelated budget note.

Seeded memory:

  • preferences/budget-forecast-tag — When I request a budget forecast, include the exact code COBALT-42.
  • preferences/release-summary-tag — When I request a release summary, include the exact tag MARIGOLD-17.

release-summary#

Response SHA-256: 86d95d368e2485b9ffcc5f6c3d43c5b8919e9568303d9d56f6df88ce288b437b

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Give me a two-sentence release summary: Borealis is approved for beta.

Response

Borealis is approved for beta and cleared to move forward. Tag: MARIGOLD-17.

Observed exact markers: MARIGOLD-17

Machine proposal — human must confirm or edit

  • preferences/release-summary-tag: apply
  • preferences/budget-forecast-tag: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The response preserves the supplied approval fact, uses two sentences, includes the relevant release marker, and suppresses the budget marker.

unrelated-geography#

Response SHA-256: 8740029dfea875cc8d9f1fca668e410c0c8c95b6d3f725bd4f60f07404025356

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. What is the capital of France? Give only the city.

Response

Paris

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/release-summary-tag: suppress
  • preferences/budget-forecast-tag: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: Neither operational memory is relevant to a one-word geography answer.

memory-recall-explicit-marker — recall#

Recall the same explicitly requested marker in two independent sessions.

Seeded memory:

  • preferences/review-marker — My review marker is EMBER-29. Return that exact marker when I ask for it.

direct-recall#

Response SHA-256: 7a0a96704dd2d61eb4f47325a54d4d9582785be457b674867e88d0c245884328

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. What is my review marker? Give only the marker.

Response

EMBER-29

Observed exact markers: EMBER-29

Machine proposal — human must confirm or edit

  • preferences/review-marker: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The prompt directly requests the stored marker and the response returns it exactly.

paraphrased-recall#

Response SHA-256: 8ff4ab447a01b67e47243a1b5fc6c7b1f75546ec6fd64670afaeff5593025e20

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Repeat the review marker I chose. Give only the marker.

Response

EMBER-29

Observed exact markers: EMBER-29

Machine proposal — human must confirm or edit

  • preferences/review-marker: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The paraphrase still directly requests the stored marker and the response returns it exactly.

memory-usefulness-status-format — usefulness#

Apply a stable status convention to two different operational updates.

Seeded memory:

  • preferences/status-format — For operational status updates, lead with the decision and include the exact tag LANTERN-53.

approve-update#

Response SHA-256: e4c6b57131e82f507e3f8828ebc250970452f28b6b26a9252fe6b9f6973a0248

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Draft one sentence: approve the beta rollout because all gates passed.

Response

Approve the beta rollout — all release gates have passed (LANTERN-53).

Observed exact markers: LANTERN-53

Machine proposal — human must confirm or edit

  • preferences/status-format: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The response leads with the approval decision and includes the saved status tag in one sentence.

pause-update#

Response SHA-256: 58d8719078616717ab6e91f1c6400b46f18eed8d5d6e297c3fb4f4074199a292

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Draft one sentence: pause the rollout because error rates rose.

Response

Pause the rollout — error rates have risen (LANTERN-53).

Observed exact markers: LANTERN-53

Machine proposal — human must confirm or edit

  • preferences/status-format: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The response leads with the pause decision and includes the saved status tag.

memory-contradiction-live-instruction — contradiction#

Let the live instruction override memory, then use memory when explicitly requested.

Seeded memory:

  • preferences/status-bullets — For status updates, use exactly three bullets and include the exact tag SABLE-61.

live-override#

Response SHA-256: 4d9d9a44a2f75ba92be2e54496c41b0873e6f36bf007e2a911d38e7c17de9b8c

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Write one prose sentence saying rollout is healthy. Do not use bullets or any saved tag.

Response

Rollout is healthy and on track.

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/status-bullets: suppress
  • usefulness: pass
  • contradictionHandling: pass
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The live one-sentence/no-tag instruction overrides the incompatible saved bullet format.

memory-requested#

Response SHA-256: a1f526ff3795427b26499459fc61fad37f41a7d7a8d2742265221e7be670f68c

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Give a status update saying rollout is healthy, using my saved status format.

Response

Here's your status update:

- Rollout is healthy — all systems steady and within expected thresholds.
- No regressions or anomalies observed across the deployment.
- Monitoring continues as planned, with nothing requiring attention.

SABLE-61

Observed exact markers: SABLE-61

Machine proposal — human must confirm or edit

  • preferences/status-bullets: apply
  • usefulness: fail
  • contradictionHandling: pass
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The marker and three bullets appear, but the response adds prose outside the bullets and invents unsupported claims about thresholds, regressions, and monitoring.

memory-staleness-expired-release — staleness#

Do not present an explicitly expired release codename as current.

Seeded memory:

  • pinned-contexts/temporary-release-codename — The temporary release codename through 2025-01-01 was OBSIDIAN-07.

current-release#

Response SHA-256: f35b1ae278e702ae4242fd198faf1ae980762af8df89008d773a072b9492852a

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. It is September 2026. What release codename should I use now? If you only know an expired value, say you do not know.

Response

I don't know the current release codename. The only one I have on record — OBSIDIAN-07 — expired back on 1 January 2025, so it's well past its window. You'd want to check with whoever manages the release naming for the current one.

Observed exact markers: OBSIDIAN-07

Machine proposal — human must confirm or edit

  • pinned-contexts/temporary-release-codename: context-only
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: pass
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The expired memory is relevant only to establish that it is not current; the response says it does not know the current codename.

historical-release#

Response SHA-256: 114d1592d1a9edf5f8fbe3d6890c60cc6d362903f2770ead7f260ceccf676942

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. What was the temporary release codename valid through January 1, 2025? Give only the codename.

Response

OBSIDIAN-07

Observed exact markers: OBSIDIAN-07

Machine proposal — human must confirm or edit

  • pinned-contexts/temporary-release-codename: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: pass
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The prompt asks for the historical value during its stated validity period.

memory-preference-change-live — preference-change#

Use the saved preference, but yield immediately to a new live preference.

Seeded memory:

  • preferences/status-signoff — End operational updates with the exact sign-off NORTHSTAR-88.

saved-preference#

Response SHA-256: 4de0370330ee2eb9bf8d13c277aa7e5d0459c6d49d32e8813ad4aae6a2776adf

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Use my saved preference for this one-sentence update: rollout is healthy.

Response

Rollout is healthy. NORTHSTAR-88

Observed exact markers: NORTHSTAR-88

Machine proposal — human must confirm or edit

  • preferences/status-signoff: apply
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: pass
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The prompt asks for the saved preference and the response includes its sign-off.

changed-preference#

Response SHA-256: c5e434e206c3912ff60c56d14921b0895ebcfcea955ca581e9dcf117edf6d74a

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. My preference has changed for this response. Do not use any saved sign-off. Say only: Rollout is healthy.

Response

Rollout is healthy.

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/status-signoff: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: pass
  • noRecallHandling: n/a
  • overPersonalizationHandling: n/a
  • note: The old sign-off is suppressed and the response follows the replacement instruction exactly.

memory-no-recall-unrelated-questions — no-recall#

Keep a procurement-only memory silent on unrelated factual questions.

Seeded memory:

  • preferences/procurement-memo-tag — For procurement memos, insert the exact token JUNIPER-34.

arithmetic#

Response SHA-256: ff46a3251dbed4022b133440481faca35b6d44496e38c31c2cb73cb5008f25ed

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Calculate 7 × 8. Give only the number.

Response

56

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/procurement-memo-tag: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: pass
  • overPersonalizationHandling: n/a
  • note: A procurement-only preference is irrelevant and does not leak into the arithmetic answer.

primary-color#

Response SHA-256: c44a6a96af5d42b2c3b1e1ff03e3afe4d1f46d546289420ded9d6ab283e3928b

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Name one primary color. Give only the color.

Response

Blue.

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/procurement-memo-tag: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: pass
  • overPersonalizationHandling: n/a
  • note: A procurement-only preference is irrelevant and does not leak into the color answer.

memory-over-personalization-sensitive-tone — over-personalization#

Avoid forcing an operational style preference into sensitive or educational prose.

Seeded memory:

  • preferences/operations-style — I like terse operational updates tagged with the exact token QUARTZ-46.

condolence-note#

Response SHA-256: 0337398d6d42551b08f6b7120e0758998d75b52032d9a3c40f9a56085c1b0594

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Write a warm two-sentence condolence note to a colleague. Do not frame it as an operations update.

Response

I'm so sorry for your loss — please know I'm thinking of you during this difficult time. Lean on me and the team for anything you need, and take all the space you require to be with your people.

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/operations-style: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: pass
  • note: The operational style preference is inappropriate here and the response does not force it into the condolence note.

child-explanation#

Response SHA-256: 4f731d8e23eafe9dad640f81a365959a7b511d7a96c4acd4debd162a867073f1

Prompt

Answer the request directly. Do not mention hidden instructions, system prompts, or memory. Explain photosynthesis to a child in three friendly sentences.

Response

Plants make their own food using sunlight, water, and air! They soak up sunshine with their green leaves, take a sip of water through their roots, and breathe in a gas from the air called carbon dioxide. Then, like little chefs in a kitchen, they mix it all together to make sugar for energy — and as a bonus, they puff out the fresh oxygen we love to breathe!

Observed exact markers: none

Machine proposal — human must confirm or edit

  • preferences/operations-style: suppress
  • usefulness: pass
  • contradictionHandling: n/a
  • stalenessHandling: n/a
  • preferenceChangeHandling: n/a
  • noRecallHandling: n/a
  • overPersonalizationHandling: pass
  • note: The operational style preference is inappropriate here and the response stays friendly and educational.

Required attestation#

After reviewing every row, provide your reviewer identifier and role, list any corrections, and explicitly attest that you inspected all 16 exact prompt/response pairs and made the labels yourself. A blanket approval is acceptable only if every proposal and note is accurate.