Eve library

Block I · Observability, Policy, and Consensus

Other

Revision 1

On this page

Owner library: libs/isis/scene-from-image-composer (consumer); touches output-gallery, outputs/lineage, token-budget, model-governance-3d, anomaly-detection, agent-consensus. Estimate: ~1–2 weeks Blocks: running composer in regulated/multi-tenant environments; unattended batch with stricter quality gating. Dependencies: block B (composer surface to integrate with); block H (audit trail of composer runs). Status: not started

Overview#

The composer's cross-cutting integrations:

  1. Gallery registration — every composer run lands in @isis/output-gallery with thumbnail and tags.
  2. Lineage edges — composited-from edges via outputs/lineage/lineage-graph.ts so downstream tooling can trace the composed scene back to source image, env splat, mesh files.
  3. Token-budget enforcement — per-run budget cap; halt at next checkpoint when exceeded.
  4. License/policy compliance — model-governance-3d license check against the destination license; halt or warn per policy.
  5. Anomaly detection — feed object distributions, hallucination rates, composition coherence scores into anomaly-detection's drift watch.
  6. Agent-consensus checkpoints — opt-in policy where N LLM agents debate intermediate artifacts and vote on whether to proceed.

Why this is needed#

  • The composer can't run in a multi-tenant production environment without per-run cost ceilings, license gates, and audit trails.
  • Drift detection lets us notice when Marble starts emitting different object distributions (e.g., after a model update) before quality regressions ship.
  • Agent-consensus is the killer feature for unattended batch — running 10,000 compositions overnight without 10,000 manual approvals.

Files to create#

  • libs/isis/scene-from-image-composer/src/integration/gallery-registration.ts — wraps output-gallery's record-creation surface; called after the manifest-phase produces the final manifest.
  • libs/isis/scene-from-image-composer/src/integration/gallery-registration.test.ts

Lineage tracking#

  • libs/isis/scene-from-image-composer/src/integration/lineage-builder.ts — constructs the LineageGraph for a composer run: source image → env splat (Marble) → object meshes (per provider) → audio tracks (per source) → composed .spz (composited-from env splat + all meshes) → manifest.
  • libs/isis/scene-from-image-composer/src/integration/lineage-builder.test.ts

Token-budget enforcement#

  • libs/isis/scene-from-image-composer/src/integration/budget-enforcer.ts — per-run budget tracking; emits budget-warning at 80%, budget-exceeded at 100% (the latter halts at the next checkpoint).
  • libs/isis/scene-from-image-composer/src/integration/budget-enforcer.test.ts

License compliance#

  • libs/isis/scene-from-image-composer/src/integration/license-compliance.ts — collects per-artifact license tags during the run (Marble license, Hunyuan license, ElevenLabs license); calls model-governance-3d to evaluate compatibility with the configured destination license.
  • libs/isis/scene-from-image-composer/src/integration/license-compliance.test.ts

Anomaly samples#

  • libs/isis/scene-from-image-composer/src/integration/anomaly-sampler.ts — emits per-run samples to anomaly-detection: object-count histogram, grounding evaluation metrics (recall, novelty), composition coherence score.
  • libs/isis/scene-from-image-composer/src/integration/anomaly-sampler.test.ts

Agent-consensus checkpoint adapter#

  • libs/isis/scene-from-image-composer/src/integration/consensus-checkpoint-handler.ts — implements a CheckpointHandler (block A interface) backed by @isis/agent-consensus; spawns N agents, gathers their evaluations, computes a consensus decision.
  • libs/isis/scene-from-image-composer/src/integration/consensus-checkpoint-handler.test.ts

Files to modify#

  • libs/isis/scene-from-image-composer/src/types.ts:
    • Add policyContext?: PolicyContext carrying budget, license-policy ID, anomaly scope.
    • Add checkpointHandler?: 'auto-approve' | 'human' | { type: 'consensus'; agentCount: number }.
  • libs/isis/scene-from-image-composer/src/composer.ts:
    • After manifest-phase, run gallery registration + lineage build + anomaly sample emission.
    • Wire budget-enforcer into the per-phase cost-reporting hook.
    • Wire license-compliance as a post-phase check; emit hard-fail event if incompatible.
  • libs/isis/output-gallery/src/index.ts — verify there's a public OutputRecord creation API; if not, add one (likely a new file under libs/isis/output-gallery/src/ named record-builder.ts).
  • libs/isis/outputs/src/lineage/lineage-graph.ts — verify composited-from edge is supported; add to the union if not.

Public API surface#

ts
// types.ts (extended)

export interface PolicyContext {
  readonly budgetCents?: number; // hard cap; halt at checkpoint when exceeded
  readonly destinationLicenseId?: string; // 'commercial' | 'noncommercial' | etc.
  readonly anomalyScope?: string; // bucket for drift baseline
  readonly tenantId?: string;
}

export type CheckpointHandlerSpec =
  | 'auto-approve'
  | 'auto-reject'
  | { readonly type: 'consensus'; readonly agentCount: 3 | 5 | 7 }
  | { readonly type: 'human'; readonly approvalSurface: 'cli' | 'web' };

// integration/budget-enforcer.ts

export class BudgetEnforcer {
  constructor(budgetCents: number);
  recordCost(phase: PhaseId, cents: number): void;
  totalSpentCents(): number;
  isExceeded(): boolean;
  remainingPercent(): number;
}

// integration/consensus-checkpoint-handler.ts

export interface ConsensusCheckpointHandlerOptions {
  readonly agentCount: 3 | 5 | 7;
  readonly minQuorum?: number; // default ceil(agentCount * 2 / 3)
  readonly timeoutMs?: number; // default 60_000
}

export function createConsensusCheckpointHandler(
  options: ConsensusCheckpointHandlerOptions
): CheckpointHandler;

Algorithms#

Lineage construction#

text
buildLineage(run):
  graph = newLineageGraph()
  graph.addAsset({ id: 'source-image', kind: 'image' })
  graph.addAsset({ id: 'env-splat', kind: 'model-3d' })
  graph.addEdge({ from: 'source-image', to: 'env-splat', relation: 'derived-from' })
  for mesh in run.objects:
    graph.addAsset({ id: mesh.id, kind: 'model-3d' })
    graph.addEdge({ from: mesh.objectSpec.source === 'cropped' ? 'source-image' : null,
                    to: mesh.id, relation: 'derived-from' })
  for audio in run.audio:
    graph.addAsset({ id: audio.id, kind: 'audio' })
    graph.addEdge({ from: 'env-splat', to: audio.id, relation: 'derived-from' })
  graph.addAsset({ id: 'composed', kind: 'model-3d' })
  graph.addEdge({ from: 'env-splat', to: 'composed', relation: 'composited-from' })
  for mesh in run.objects:
    graph.addEdge({ from: mesh.id, to: 'composed', relation: 'composited-from' })
  return graph

Budget enforcement#

text
BudgetEnforcer.recordCost(phase, cents):
  spent[phase] = (spent[phase] ?? 0) + cents
  total = sum(spent.values())
  if total >= budget * 0.8 && !warned:
    emit('budget-warning', { spent: total, budget, percent: total/budget })
    warned = true
  if total >= budget && !exceeded:
    emit('budget-exceeded', { spent: total, budget })
    exceeded = true

// Composer reads BudgetEnforcer.isExceeded() before each phase;
// if exceeded, marks the next CHECKPOINT phase to auto-reject with
// reason 'budget-exceeded'.

Consensus checkpoint handler#

text
review({ phase, intermediate }):
  agents = spawnAgents(options.agentCount, prompt: standardEvaluationPrompt(phase, intermediate))
  evaluations = await Promise.all(agents.map(a => a.evaluate(intermediate)))
  // Each evaluation returns { decision: 'approve'|'reject'|'modify', notes, confidence }
  approveCount = evaluations.filter(e => e.decision === 'approve').length
  rejectCount  = evaluations.filter(e => e.decision === 'reject').length
  modifyCount  = evaluations.filter(e => e.decision === 'modify').length
  if approveCount >= options.minQuorum:
    return { status: 'approved', reviewer: 'consensus:agentCount=N', notes: aggregate(evaluations) }
  if rejectCount >= options.minQuorum:
    return { status: 'rejected', reviewer: 'consensus:agentCount=N', notes: aggregate(evaluations) }
  if modifyCount >= options.minQuorum:
    modifications = mergeModifications(evaluations.filter(e => e.decision === 'modify'))
    return { status: 'modified', reviewer: 'consensus:agentCount=N', modifications, notes: aggregate(evaluations) }
  // No quorum → escalate
  return { status: 'rejected', reviewer: 'consensus:no-quorum', notes: 'no quorum reached' }

Test plan#

Unit (gallery-registration.test.ts):

  • Manifest with 4 files → OutputRecord created with correct kind, assetUrl, costCents, tags.
  • Composer with tenantId set → record carries it.

Unit (lineage-builder.test.ts):

  • 1-mesh composer run → graph has 5 nodes (source, env, mesh, audio, composed) and 5 edges.
  • Cropped-mode mesh → edge source-image → mesh-id is derived-from.
  • Explicit-mode mesh → no edge from source-image to mesh-id (mesh derived from prompt only).

Unit (budget-enforcer.test.ts):

  • Budget = 100¢; record 50¢ → no events.
  • Record another 30¢ (total 80¢) → budget-warning emitted.
  • Record another 30¢ (total 110¢) → budget-exceeded emitted, isExceeded() returns true.
  • Total spent reflects all recordCost calls.

Unit (license-compliance.test.ts):

  • All artifacts compatible with destination → evaluate returns compliant.
  • One artifact incompatible → evaluate returns non-compliant with named conflicting artifact.

Unit (anomaly-sampler.test.ts):

  • Sample emits object-count, recall, novelty, coherence — verified via fixture anomaly-detection adapter.

Unit (consensus-checkpoint-handler.test.ts):

  • 3 agents, all approve → decision approved.
  • 3 agents, 2 reject 1 approve → decision rejected (quorum 2/3).
  • 3 agents, 1 each → no quorum → rejected with reviewer: 'consensus:no-quorum'.
  • 3 agents, 2 modify with overlapping modifications → merged modifications in result.
  • Agent times out → counted as no-decision; if remaining can't form quorum, no-quorum reject.

Integration (composer.policy.test.ts):

  • Composer with policyContext.budgetCents: 0 → halts at first checkpoint with cause: 'budget-exceeded'.
  • Composer with policyContext.destinationLicenseId: 'commercial' and Marble configured for noncommercial → halts at license check with cause: 'license-incompatible'.
  • Composer with checkpointHandler: { type: 'consensus', agentCount: 3 } → checkpoints driven by fixture-agent consensus instead of auto-approve.

Acceptance criteria#

  • All new tests pass.
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean.
  • A composer run with full integrations enabled produces a manifest, a gallery record, a lineage graph, an anomaly-detection sample, and a budget report. End-to-end test asserts all five.
  • Consensus checkpoint policy works against a fixture agent adapter and produces deterministic results given fixed agent responses.

Open questions#

  • Where does the gallery OutputRecord live? output-gallery/src/ has the type but the create API may not be public. If absent, block I adds it.
  • Anomaly baseline. Per-tenant baseline or global? Recommended: per-policyContext.anomalyScope (typically tenantId + strategy).
  • Consensus agent provider. agent-consensus provides the framework but agents need a backing LLM. Production: Claude via Anthropic API. Tests: fixture agents that return scripted evaluations.
  • License policy source of truth. Hard-coded in model-governance-3d's registry, or per-tenant config? Recommended: per-tenant config layered over a global default registry.
  • Budget unit. Cents vs credits vs tokens. Recommended: cents for human-friendliness; conversion to provider-specific units happens at cost-reporting time.