Eve library

@isis/3d-browser

Other

Revision 1

On this page

Browser-first 3D viewing, gaussian splatting, editing, and inference infrastructure for Isis.

Purpose#

@isis/3d-browser is the Phase 71 browser surface for generated 3D assets. The project scaffold is intentionally split into four future-facing modules so the remaining 71.5.x tasks can land without reorganizing the library:

  • viewer: mesh viewing, PBR inspection, comparison, and progressive streaming
  • gaussian-splatting: WebGPU 3DGS and 4DGS playback
  • editing: mesh sculpting, painting, UV inspection, history, and export
  • inference: browser-side WASM/WebGPU inference and hybrid offload

Implemented Viewer Surface#

71.5.1.2, 71.5.1.3, 71.5.1.4, 71.5.1.5, 71.5.1.6, 71.5.1.7, 71.5.1.8, 71.5.1.9, and 71.5.1.10 are now implemented through the WebGPU-native PBR renderer in src/viewer/pbr-renderer.ts, the projected wireframe helper in src/viewer/wireframe-overlay.ts, the UV atlas helper in src/viewer/uv-overlay.ts, and the turntable interaction surface in src/viewer/turntable-viewer.ts, plus the reusable scene analysis helper in src/viewer/mesh-statistics.ts, the comparison snapshot helper in src/viewer/comparison-mode.ts, the AR Quick Look launch helper in src/viewer/quick-look.ts, and the progressive LOD-streaming helper in src/viewer/progressive-streaming.ts. Together they provide:

  • physically based direct lighting with GGX shading
  • diffuse SH environment lighting and prefiltered reflection cube sampling
  • directional shadow mapping with PCF filtering and normal bias
  • typed geometry, material, environment, and camera contracts for generated 3D assets
  • deterministic CPU-side uniform packing and cache-aware resource preparation
  • scene-wide PBR inspection modes for lit, albedo, normal, roughness, and metallic review without leaving the browser viewer
  • reusable scene statistics for polygon, vertex, material, texture, bounds, and approximate resource-weight review
  • headless comparison contracts for single, side-by-side, and overlay review of two generated variants using shared scene statistics and delta summaries
  • typed AR Quick Look launch-url generation and browser capability assessment for same-origin USDZ review flows on Safari
  • typed progressive LOD manifests, screen-coverage selection, and async preview-to-full-quality mesh promotion for browser streaming workflows
  • orbit, pan, wheel zoom, fit-to-scene framing, and optional auto-turntable camera control
  • reusable UV atlas extraction with island, triangle, coverage, and bounds metadata for browser-side layout inspection
  • indexed-triangle wireframe projection for mesh topology inspection in browser
  • viewer orchestration that can drive renderer-backed frames or DOM-hosted QA harnesses from the same camera model

Implemented Gaussian Splatting Surface#

71.5.2.1, 71.5.2.2, 71.5.2.3, 71.5.2.4, 71.5.2.5, and 71.5.2.6 are now implemented through the WebGPU 3D Gaussian Splatting surface in src/gaussian-splatting/renderer.ts, the shared scene loader in src/gaussian-splatting/scene-loader.ts, the shared LOD helper in src/gaussian-splatting/lod.ts, the .igsplat transport helper in src/gaussian-splatting/scene-compression.ts, the shared mesh extraction helper in src/gaussian-splatting/mesh-extraction.ts, the shared animation helper in src/gaussian-splatting/animation.ts, and the Oshun studio route host in apps/oshun/web/src/components/studio/StudioIsisGaussianSplattingWorkspace.tsx. Together they provide:

  • typed scene, camera, sort, and frame-profile contracts for browser 3DGS review
  • typed .ply and .splat scene loading contracts with format detection and URL-fetch helpers
  • deterministic scene validation and GPU packing for positions, scales, rotations, colors, and opacities
  • GraphDeco-style PLY parsing for ASCII and binary little-endian payloads, including SH/DC color decode, opacity sigmoid decode, and retained spherical-harmonics coefficients up to degree 3
  • .splat parsing for both raw 32-byte standard payloads and the repo worker's ISIS_SPLAT_V1 headered binary payloads
  • typed .igsplat transport compression and decompression with quantized position, scale, rotation, color, opacity, and SH payload packing plus optional raw, gzip, or brotli transport codecs
  • checksum-validated round-trip decoding and URL fetch helpers for browser-side network transfer review
  • deterministic browser-side LOD generation with bounds-preserving anchors, selectable reduction strategies, and hysteresis-backed coverage bands
  • browser-side 4D gaussian clip validation, interpolation, stable LOD-aware clip subsetting, and reusable playback-state control for animated scene review
  • browser-side gaussian-to-mesh extraction through density voxelization, tetrahedral isosurface recovery, smoothing passes, and mesh-scene conversion
  • extracted mesh statistics and wireframe-preview surfaces that can feed browser inspection routes without leaving the gaussian workspace
  • cull-before-sort visibility rejection using clip-space radius estimates
  • CPU-native and depth-bucket sort strategies for browser-friendly depth ordering
  • WebGPU billboard expansion that projects rotated Gaussian axes directly in WGSL
  • premultiplied-alpha Gaussian falloff composition with reusable scene buffers
  • stable sort reuse across small camera deltas to avoid redundant upload work
  • browser-route telemetry for ingest status, parser metadata, active LOD level, 4D playback timing, transfer compression status, coverage, culling, and payload size

Implemented Editing Surface#

71.5.3.1, 71.5.3.2, 71.5.3.3, 71.5.3.4, 71.5.3.5, 71.5.3.6, 71.5.3.7, and 71.5.3.8 are now implemented through the shared sculpting surface in src/editing/sculpting.ts, the shared UV-aware texture painting surface in src/editing/texture-painting.ts, the shared vertex-painting surface in src/editing/vertex-painting.ts, the shared boolean surface in src/editing/boolean-operations.ts, the shared UV layout surface in src/editing/uv-layout.ts, the shared bounded history surface in src/editing/history.ts, the shared export surface in src/editing/mesh-export.ts, and the Oshun route host in apps/oshun/web/src/components/studio/StudioIsis3DEditingWorkspace.tsx. Together they provide:

  • typed browser sculpting contracts for push, pull, smooth, and flatten brushes
  • typed texture-painting contracts for UV-aware world-space strokes, blend modes, upload planning, and sampled texture inspection
  • typed vertex-painting contracts for per-vertex material assignment, blend modes, interleaved upload planning, and weighted influence inspection
  • typed browser boolean contracts for union, subtract, and intersect against sphere, box, and cylinder operands with deterministic relief-volume sampling and preserved mesh topology
  • typed browser UV layout contracts for atlas inspection, packed unwrap preview, manual island translation, scale, rotation, snapping, and texcoord upload telemetry
  • typed bounded snapshot history contracts for immutable geometry/texture checkpoints, undo depth, redo depth, and reset checkpoints
  • reusable geometry validation and deterministic topology adjacency generation
  • stroke preview sampling with weighted influence metadata and affected-vertex accounting
  • UV-raster paint preview sampling with affected-texel, coverage, and atlas bounds metadata
  • boolean preview sampling with per-vertex surface deltas, affected-triangle counts, and estimated volume-change telemetry
  • CPU-backed deformation application that preserves shared mesh contracts for browser QA and fallback runtimes
  • CPU-backed texture rasterization that paints directly onto browser-resident base-color maps and publishes WebGPU upload-ready layout telemetry
  • CPU-backed vertex color painting that updates browser-resident geometry color layers and publishes interleaved vertex-buffer upload telemetry
  • CPU-backed UV layout inspection that reports island bounds, texel density, overlap, out-of-bounds coverage, and packed atlas utilization from any mesh
  • CPU-backed UV layout application that commits manual texcoord transforms back into browser-resident geometry and emits WebGPU upload-ready stride and byte telemetry
  • CPU-backed constructive relief booleans that update browser-resident mesh positions and normals while preserving UVs, colors, and triangle topology for downstream sculpt, paint, and export workflows
  • immutable geometry and texture snapshot cloning for browser-side edit-history branching, replay, and reset restoration
  • typed multi-format mesh export for GLB, inline glTF, OBJ bundles, FBX bundles, USDA bundles, and USDZ packages emitted directly from the browser editing state
  • deterministic PNG texture encoding and stored-zip packaging utilities for portable browser-side export bundles
  • glTF-native export with embedded PBR texture payloads plus OBJ, FBX, and USD interchange handoff documents that preserve the active mesh, UVs, normals, vertex colors, and base-color surface
  • per-stroke normal recomputation and bounds/displacement telemetry
  • WebGPU compute shader source generation and dispatch sizing for brush and normal reconstruction passes
  • a dedicated studio workspace route with shared sculpt, texture-paint, UV-layout, vertex-paint, boolean, and undo/redo history mode switching, live inspection surfaces, preset-driven QA targets, keyboard history shortcuts, shader or upload telemetry, direct multi-format export downloads, and route discovery

Implemented Inference Surface#

71.5.4.1, 71.5.4.2, 71.5.4.3, 71.5.4.4, and 71.5.4.5 are now implemented through the shared browser SF3D WASM surface in src/inference/sf3d-wasm.ts, the shared browser TripoSR WASM surface in src/inference/triposr-wasm.ts, the generic ONNX WebGPU backend in src/inference/webgpu-backend.ts, the browser model cache in src/inference/model-cache.ts, the hybrid browser or server orchestrator in src/inference/hybrid-inference.ts, and the module export surface in src/inference/index.ts. Together they provide:

  • typed browser capability assessment for secure-context, SIMD, threaded WASM, SharedArrayBuffer, and IndexedDB-aware SF3D execution planning
  • typed bundle manifests for ONNX Runtime Web loaders, WASM binaries, and quantized SF3D ORT or ONNX graph artifacts
  • deterministic browser compile planning for draft, standard, and high-quality image-to-3D presets with runtime-variant, memory, latency, and artifact selection telemetry
  • foreground-aware RGBA preprocessing that crops around the alpha silhouette, resizes to the selected browser inference resolution, and emits coverage, luma, and fill-ratio metrics
  • an embedded browser-loadable WASM feature kernel that runs without server dependencies and exposes a stable depth-byte ABI for mesh synthesis
  • browser-side single-image relief-mesh synthesis that emits closed triangle geometry, recomputed normals, vertex colors, UVs, and a PBR-ready base-color texture handoff
  • a lighter TripoSR browser compile profile with smaller target resolutions, faster estimated latency budgets, and explicit vertex-color versus baked- texture output planning
  • browser-side TripoSR mesh synthesis that defaults to vertex-color output for smaller client payloads while still supporting optional baked-texture handoffs that mirror the upstream model surface
  • typed ONNX Runtime WebGPU backend contracts for capability assessment, session creation, tensor marshaling, graph-optimization selection, and GPU-buffer output preferences
  • reusable WebGPU execution-provider planning that can bind quantized ONNX or ORT models through the browser webgpu execution provider without forcing a hard dependency on onnxruntime-web into the shared package surface
  • reusable IndexedDB-backed browser model caching with explicit artifact bundle contracts, LRU eviction, per-model inspection, and one-time download reuse across browser sessions
  • reusable artifact acquisition flows that can prefetch selected SF3D or TripoSR runtime artifacts, resolve browser-facing asset URLs, and materialize cached buffers back into inference sessions without redesigning the inference module surface
  • reusable hybrid image-to-3D planning that weighs browser latency, device memory, network upload cost, remote queue delay, and requested quality to choose browser-only, server-only, or preview-plus-offload execution
  • a browser-side client for the repo's /v1/3d provider contract, including capability discovery, image-to-3D job submission, polling, cancellation, and normalized remote asset descriptors for final GLB or OBJ handoff
  • unified hybrid orchestration that can emit a fast local preview, offload the final asset to a remote 3D provider when browser execution is constrained, and publish cross-stage progress updates without changing the surrounding app surface

Frontend Direction#

  • Visual thesis: a cinematic, studio-grade browser workspace with the canvas as the primary surface and controls treated as supporting instrumentation.
  • Content plan: immersive model viewport, focused inspection rail, editing and gaussian workflows, export and publish handoff.
  • Interaction thesis: smooth orbit-first camera motion, intentional inspection overlays, and progressive reveal from low-latency preview to full-fidelity asset.

Verification#

bash
NX_DAEMON=false pnpm nx test @isis/3d-browser
NX_DAEMON=false pnpm nx lint @isis/3d-browser
NX_DAEMON=false pnpm nx build @isis/3d-browser
pnpm tsc -p libs/isis/3d-browser/tsconfig.json --noEmit