Browser-first 3D viewing, gaussian splatting, editing, and inference infrastructure for Isis.
Purpose#
@isis/3d-browser is the Phase 71 browser surface for generated 3D assets. The
project scaffold is intentionally split into four future-facing modules so the
remaining 71.5.x tasks can land without reorganizing the library:
viewer: mesh viewing, PBR inspection, comparison, and progressive streaminggaussian-splatting: WebGPU 3DGS and 4DGS playbackediting: mesh sculpting, painting, UV inspection, history, and exportinference: browser-side WASM/WebGPU inference and hybrid offload
Implemented Viewer Surface#
71.5.1.2, 71.5.1.3, 71.5.1.4, 71.5.1.5, 71.5.1.6, 71.5.1.7,
71.5.1.8, 71.5.1.9, and 71.5.1.10 are now implemented through the
WebGPU-native PBR renderer in src/viewer/pbr-renderer.ts, the projected
wireframe helper in src/viewer/wireframe-overlay.ts, the UV atlas helper in
src/viewer/uv-overlay.ts, and the turntable interaction surface in
src/viewer/turntable-viewer.ts, plus the reusable scene analysis helper in
src/viewer/mesh-statistics.ts, the comparison snapshot helper in
src/viewer/comparison-mode.ts, the AR Quick Look launch helper in
src/viewer/quick-look.ts, and the progressive LOD-streaming helper in
src/viewer/progressive-streaming.ts. Together they provide:
- physically based direct lighting with GGX shading
- diffuse SH environment lighting and prefiltered reflection cube sampling
- directional shadow mapping with PCF filtering and normal bias
- typed geometry, material, environment, and camera contracts for generated 3D assets
- deterministic CPU-side uniform packing and cache-aware resource preparation
- scene-wide PBR inspection modes for lit, albedo, normal, roughness, and metallic review without leaving the browser viewer
- reusable scene statistics for polygon, vertex, material, texture, bounds, and approximate resource-weight review
- headless comparison contracts for single, side-by-side, and overlay review of two generated variants using shared scene statistics and delta summaries
- typed AR Quick Look launch-url generation and browser capability assessment for same-origin USDZ review flows on Safari
- typed progressive LOD manifests, screen-coverage selection, and async preview-to-full-quality mesh promotion for browser streaming workflows
- orbit, pan, wheel zoom, fit-to-scene framing, and optional auto-turntable camera control
- reusable UV atlas extraction with island, triangle, coverage, and bounds metadata for browser-side layout inspection
- indexed-triangle wireframe projection for mesh topology inspection in browser
- viewer orchestration that can drive renderer-backed frames or DOM-hosted QA harnesses from the same camera model
Implemented Gaussian Splatting Surface#
71.5.2.1, 71.5.2.2, 71.5.2.3, 71.5.2.4, 71.5.2.5, and 71.5.2.6 are
now implemented through the WebGPU 3D Gaussian Splatting surface in
src/gaussian-splatting/renderer.ts, the shared scene loader in
src/gaussian-splatting/scene-loader.ts, the shared LOD helper in
src/gaussian-splatting/lod.ts, the .igsplat transport helper in
src/gaussian-splatting/scene-compression.ts, the shared mesh extraction helper
in src/gaussian-splatting/mesh-extraction.ts, the shared animation helper in
src/gaussian-splatting/animation.ts, and the Oshun studio route host in
apps/oshun/web/src/components/studio/StudioIsisGaussianSplattingWorkspace.tsx.
Together they provide:
- typed scene, camera, sort, and frame-profile contracts for browser 3DGS review
- typed
.plyand.splatscene loading contracts with format detection and URL-fetch helpers - deterministic scene validation and GPU packing for positions, scales, rotations, colors, and opacities
- GraphDeco-style PLY parsing for ASCII and binary little-endian payloads, including SH/DC color decode, opacity sigmoid decode, and retained spherical-harmonics coefficients up to degree 3
.splatparsing for both raw 32-byte standard payloads and the repo worker'sISIS_SPLAT_V1headered binary payloads- typed
.igsplattransport compression and decompression with quantized position, scale, rotation, color, opacity, and SH payload packing plus optional raw, gzip, or brotli transport codecs - checksum-validated round-trip decoding and URL fetch helpers for browser-side network transfer review
- deterministic browser-side LOD generation with bounds-preserving anchors, selectable reduction strategies, and hysteresis-backed coverage bands
- browser-side 4D gaussian clip validation, interpolation, stable LOD-aware clip subsetting, and reusable playback-state control for animated scene review
- browser-side gaussian-to-mesh extraction through density voxelization, tetrahedral isosurface recovery, smoothing passes, and mesh-scene conversion
- extracted mesh statistics and wireframe-preview surfaces that can feed browser inspection routes without leaving the gaussian workspace
- cull-before-sort visibility rejection using clip-space radius estimates
- CPU-native and depth-bucket sort strategies for browser-friendly depth ordering
- WebGPU billboard expansion that projects rotated Gaussian axes directly in WGSL
- premultiplied-alpha Gaussian falloff composition with reusable scene buffers
- stable sort reuse across small camera deltas to avoid redundant upload work
- browser-route telemetry for ingest status, parser metadata, active LOD level, 4D playback timing, transfer compression status, coverage, culling, and payload size
Implemented Editing Surface#
71.5.3.1, 71.5.3.2, 71.5.3.3, 71.5.3.4, 71.5.3.5, 71.5.3.6,
71.5.3.7, and 71.5.3.8 are now implemented through the shared sculpting
surface in src/editing/sculpting.ts, the shared UV-aware texture painting
surface in src/editing/texture-painting.ts, the shared vertex-painting surface
in src/editing/vertex-painting.ts, the shared boolean surface in
src/editing/boolean-operations.ts, the shared UV layout surface in
src/editing/uv-layout.ts, the shared bounded history surface in
src/editing/history.ts, the shared export surface in
src/editing/mesh-export.ts, and the Oshun route host in
apps/oshun/web/src/components/studio/StudioIsis3DEditingWorkspace.tsx.
Together they provide:
- typed browser sculpting contracts for push, pull, smooth, and flatten brushes
- typed texture-painting contracts for UV-aware world-space strokes, blend modes, upload planning, and sampled texture inspection
- typed vertex-painting contracts for per-vertex material assignment, blend modes, interleaved upload planning, and weighted influence inspection
- typed browser boolean contracts for union, subtract, and intersect against sphere, box, and cylinder operands with deterministic relief-volume sampling and preserved mesh topology
- typed browser UV layout contracts for atlas inspection, packed unwrap preview, manual island translation, scale, rotation, snapping, and texcoord upload telemetry
- typed bounded snapshot history contracts for immutable geometry/texture checkpoints, undo depth, redo depth, and reset checkpoints
- reusable geometry validation and deterministic topology adjacency generation
- stroke preview sampling with weighted influence metadata and affected-vertex accounting
- UV-raster paint preview sampling with affected-texel, coverage, and atlas bounds metadata
- boolean preview sampling with per-vertex surface deltas, affected-triangle counts, and estimated volume-change telemetry
- CPU-backed deformation application that preserves shared mesh contracts for browser QA and fallback runtimes
- CPU-backed texture rasterization that paints directly onto browser-resident base-color maps and publishes WebGPU upload-ready layout telemetry
- CPU-backed vertex color painting that updates browser-resident geometry color layers and publishes interleaved vertex-buffer upload telemetry
- CPU-backed UV layout inspection that reports island bounds, texel density, overlap, out-of-bounds coverage, and packed atlas utilization from any mesh
- CPU-backed UV layout application that commits manual texcoord transforms back into browser-resident geometry and emits WebGPU upload-ready stride and byte telemetry
- CPU-backed constructive relief booleans that update browser-resident mesh positions and normals while preserving UVs, colors, and triangle topology for downstream sculpt, paint, and export workflows
- immutable geometry and texture snapshot cloning for browser-side edit-history branching, replay, and reset restoration
- typed multi-format mesh export for GLB, inline glTF, OBJ bundles, FBX bundles, USDA bundles, and USDZ packages emitted directly from the browser editing state
- deterministic PNG texture encoding and stored-zip packaging utilities for portable browser-side export bundles
- glTF-native export with embedded PBR texture payloads plus OBJ, FBX, and USD interchange handoff documents that preserve the active mesh, UVs, normals, vertex colors, and base-color surface
- per-stroke normal recomputation and bounds/displacement telemetry
- WebGPU compute shader source generation and dispatch sizing for brush and normal reconstruction passes
- a dedicated studio workspace route with shared sculpt, texture-paint, UV-layout, vertex-paint, boolean, and undo/redo history mode switching, live inspection surfaces, preset-driven QA targets, keyboard history shortcuts, shader or upload telemetry, direct multi-format export downloads, and route discovery
Implemented Inference Surface#
71.5.4.1, 71.5.4.2, 71.5.4.3, 71.5.4.4, and 71.5.4.5 are now
implemented through the shared browser SF3D WASM surface in
src/inference/sf3d-wasm.ts, the shared browser TripoSR WASM surface in
src/inference/triposr-wasm.ts, the generic ONNX WebGPU backend in
src/inference/webgpu-backend.ts, the browser model cache in
src/inference/model-cache.ts, the hybrid browser or server orchestrator in
src/inference/hybrid-inference.ts, and the module export surface in
src/inference/index.ts. Together they provide:
- typed browser capability assessment for secure-context, SIMD, threaded WASM, SharedArrayBuffer, and IndexedDB-aware SF3D execution planning
- typed bundle manifests for ONNX Runtime Web loaders, WASM binaries, and quantized SF3D ORT or ONNX graph artifacts
- deterministic browser compile planning for draft, standard, and high-quality image-to-3D presets with runtime-variant, memory, latency, and artifact selection telemetry
- foreground-aware RGBA preprocessing that crops around the alpha silhouette, resizes to the selected browser inference resolution, and emits coverage, luma, and fill-ratio metrics
- an embedded browser-loadable WASM feature kernel that runs without server dependencies and exposes a stable depth-byte ABI for mesh synthesis
- browser-side single-image relief-mesh synthesis that emits closed triangle geometry, recomputed normals, vertex colors, UVs, and a PBR-ready base-color texture handoff
- a lighter TripoSR browser compile profile with smaller target resolutions, faster estimated latency budgets, and explicit vertex-color versus baked- texture output planning
- browser-side TripoSR mesh synthesis that defaults to vertex-color output for smaller client payloads while still supporting optional baked-texture handoffs that mirror the upstream model surface
- typed ONNX Runtime WebGPU backend contracts for capability assessment, session creation, tensor marshaling, graph-optimization selection, and GPU-buffer output preferences
- reusable WebGPU execution-provider planning that can bind quantized ONNX or
ORT models through the browser
webgpuexecution provider without forcing a hard dependency ononnxruntime-webinto the shared package surface - reusable IndexedDB-backed browser model caching with explicit artifact bundle contracts, LRU eviction, per-model inspection, and one-time download reuse across browser sessions
- reusable artifact acquisition flows that can prefetch selected SF3D or TripoSR runtime artifacts, resolve browser-facing asset URLs, and materialize cached buffers back into inference sessions without redesigning the inference module surface
- reusable hybrid image-to-3D planning that weighs browser latency, device memory, network upload cost, remote queue delay, and requested quality to choose browser-only, server-only, or preview-plus-offload execution
- a browser-side client for the repo's
/v1/3dprovider contract, including capability discovery, image-to-3D job submission, polling, cancellation, and normalized remote asset descriptors for final GLB or OBJ handoff - unified hybrid orchestration that can emit a fast local preview, offload the final asset to a remote 3D provider when browser execution is constrained, and publish cross-stage progress updates without changing the surrounding app surface
Frontend Direction#
- Visual thesis: a cinematic, studio-grade browser workspace with the canvas as the primary surface and controls treated as supporting instrumentation.
- Content plan: immersive model viewport, focused inspection rail, editing and gaussian workflows, export and publish handoff.
- Interaction thesis: smooth orbit-first camera motion, intentional inspection overlays, and progressive reveal from low-latency preview to full-fidelity asset.
Verification#
NX_DAEMON=false pnpm nx test @isis/3d-browser
NX_DAEMON=false pnpm nx lint @isis/3d-browser
NX_DAEMON=false pnpm nx build @isis/3d-browser
pnpm tsc -p libs/isis/3d-browser/tsconfig.json --noEmit