Eve library

@isis/3d-generation-benchmarks

Other

Revision 1

On this page

Benchmarking domain library for Phase 71 3D generation quality validation.

Purpose#

This package establishes the dedicated home for Isis 3D benchmark infrastructure, quality metrics, and competitive comparison workflows. The initial foundation ships with a standalone seed dataset and threshold-gated runner so the package is immediately executable while later Phase 71 tasks add larger prompt/image corpora, richer metrics, and external provider comparisons. It now also includes an automated provider/model execution runner that can drive the standardized prompt and reference-image suites across existing @isis/3d-generation providers and collect per-run output reports. Those reports can also be stored in the package's versioned benchmark history store so quality trends remain queryable as providers and model revisions change over time.

Modules#

  • ./infrastructure: benchmark suite orchestration, seed dataset coverage, and the report contract plus standardized prompt/reference-image corpora and the automated provider/model generation matrix runner plus versioned benchmark result storage/history.
  • ./metrics: blueprint surface for geometric, topology, texture, adherence, and consistency scoring work planned in 71.7.2.x.
  • ./comparison: blueprint surface for external benchmark adapters, rendering, reports, and leaderboard work planned in 71.7.3.x.

Verification#

bash
NX_DAEMON=false pnpm nx test @isis/3d-generation-benchmarks
NX_DAEMON=false pnpm nx lint @isis/3d-generation-benchmarks
NX_DAEMON=false pnpm nx build @isis/3d-generation-benchmarks
pnpm tsc -p libs/isis/3d-generation-benchmarks/tsconfig.json --noEmit