Eve library

@isis/3d-inference-local

Other

Revision 1

On this page

Self-hosted 3D inference primitives for the Isis domain.

This library is the local runtime kernel for Phase 71. It provides:

  • Nx project metadata and build/test/lint wiring
  • strongly typed contracts for local 3D model descriptors, GPU server config, queueing, warm-up, and lifecycle management
  • provider interoperability helpers so self-hosted models can plug into the existing @isis/3d-generation provider surface
  • disk-aware model weight caching with access tracking, cache inspection, and LRU eviction for large local model installs
  • per-GPU VRAM budget management with pressure detection and automatic model offloading hooks for local multi-model serving
  • capability-aware multi-model scheduling that selects, loads, reuses, and unloads text-to-3D, image-to-3D, and texturing models based on incoming request mix
  • a starvation-aware request queue with interactive, batch, and background priority classes to prepare later batching and multi-GPU dispatch
  • a batch aggregation layer that groups queue-compatible 3D requests into throughput-oriented activation batches using a configurable aggregation window
  • a multi-GPU load balancer that routes requests across VRAM-aware schedulers while honoring soft affinity and hard per-model GPU pinning
  • explicit VRAM-tier placement that keeps ~6GB models on consumer GPUs, routes 24GB-class workloads onto A100/H100-style accelerators when available, and skips GPUs that can never satisfy a model's VRAM floor
  • a model prewarm manager that keeps hot models resident in VRAM, demotes cold models into disk-backed offloaded state for fast reloads, and plugs directly into the existing warmup-handler surface on the local GPU model server
  • a queue-depth autoscaler that watches 3D generation backlog, chooses the highest-priority RunPod inference/shared pool with spare headroom, and emits threshold/cooldown-aware scale-up requests through a provisioner callback
  • spot/preemptible-capacity-aware pool routing so batch/background 3D jobs prefer cheaper interruptible GPU pools with automatic on-demand fallback while interactive traffic stays on stable on-demand capacity by default
  • a GPU utilization monitor that correlates time-windowed utilization samples with scheduler residency, raises lifecycle-aware alerts for underutilized GPUs and idle resident model instances, and exposes poll/reconcile APIs for external alert delivery
  • a GPU node-pool manager that keeps 3D inference capacity isolated from training workloads, tracks Kubernetes/RunPod/manual pool membership, and exposes pool-aware node selection plus heartbeat-backed capacity snapshots
  • Meshy-compatible SSE progress streams with resumable Last-Event-ID replay, task-scoped stream aliases, and typed progress publishing APIs
  • a unified provider-oriented REST API for capabilities, text/image job submission, status polling, cancellation, per-job streams, and result asset downloads
  • package-local CUDA container assets for NVIDIA Container Toolkit deployment, including a Docker build, compose stack, and env-driven runtime bootstrap
  • a package-local Helm chart for Kubernetes GPU deployments with NVIDIA node selectors, GPU resource requests/limits, persistent model/cache volumes, and optional runtime-module mounts
  • RunPod serverless adapters that proxy the local Isis-compatible REST/SSE API onto RunPod /run, /runsync, /status, /cancel, and /health endpoints for scale-to-zero operation
  • a built-in cost dashboard that tracks self-hosted cost per asset and compares it against configurable cloud API pricing profiles such as Meshy plans
  • a dependency-aware TRELLIS.2 loader that registers the official Microsoft checkpoints, resolves the cross-repo sparse-structure decoder dependency, and produces bf16/fp16 runtime load plans with low-VRAM guidance
  • a runnable TRELLIS text-to-3D mesh pipeline that patches the official Microsoft text-conditioned checkpoint bundle into a fully local runtime bundle, executes the upstream Python pipeline, and emits staged progress for text conditioning, sparse-structure sampling, SLat sampling, and mesh extraction
  • a runnable TRELLIS.2 single-image reconstruction pipeline that patches the official Microsoft image-conditioned checkpoint bundle into a fully local runtime bundle and emits staged progress for preprocessing, image conditioning, sparse-structure sampling, shape SLat sampling, and mesh extraction
  • a runnable TRELLIS.2 multi-image reconstruction pipeline that accepts 2-6 reference views, fuses the strongest views into one conditioning sequence, and returns raw mesh output with explicit selected-view metadata
  • a runnable TRELLIS.2 PBR extraction pipeline that bakes albedo, metallic, roughness, alpha, and combined metallic-roughness textures from the official TRELLIS.2 texturing pipeline
  • a runnable TRELLIS.2 Gaussian splatting pipeline that projects TRELLIS.2 mesh+appearance outputs into GraphDECO-compatible .ply splats for real-time viewing workflows
  • a runnable TRELLIS.2 GLB/GLTF export pipeline that packages textured meshes as single-file GLB or embedded-resource GLTF assets
  • preset-driven TRELLIS.2 quality controls that map draft, standard, high, and ultra onto released pipeline resolutions, extraction grids, and token/step budgets
  • a TRELLIS.2 prompt adapter that converts Isis prompt templates into object-centric reference-image prompts and structured guidance metadata for the released image-conditioned runtime
  • deterministic TRELLIS.2 seed control with normalized replayable seeds across mesh, texturing, GLTF, and Gaussian export paths
  • TRELLIS.2 negative prompt support that converts Isis exclusions into upstream reference-image avoidance guidance for the released image-conditioned runtime
  • TRELLIS.2 style conditioning presets for photorealistic, stylized, low-poly, and cartoon prompt-to-image guidance
  • a deterministic TRELLIS.2 benchmark suite that scores local quality against Meshy 6, Rodin Gen-1.5, and Tripo v2.5 commercial baselines with weighted case metrics and threshold gates
  • TRELLIS.2 update automation that polls Microsoft GitHub, falls back from releases to tags to branch-head commits when needed, downloads fresh runtime source archives, and syncs the paired Hugging Face checkpoints
  • a Hunyuan3D 2.1 shape loader that registers Tencent’s official tencent/Hunyuan3D-2.1 checkpoint bundle, restricts downloads to the shape subfolder, and resolves standard, low-VRAM, and 6GB mmgp load plans with offline runtime hints
  • a Hunyuan3D 2.1 MMGP coordinator that stages text, shape, paint, super-resolution, and bake phases through the existing GPU memory manager so 6GB-class consumer GPUs can execute the published Hunyuan flows sequentially
  • a Hunyuan3D 2.1 Blender addon integration that installs as a single .py addon, submits text/image jobs through the local provider API, polls job status, and imports completed models back into the active Blender scene
  • a Hunyuan3D 2.1 text-to-shape pipeline that follows Tencent’s published flow of HunyuanDiT prompt-to-image generation, optional background removal, and local Hunyuan3D 2.1 shape reconstruction with preview-image capture
  • a Hunyuan3D 2.1 image-to-shape pipeline for direct image-conditioned shape reconstruction, including background-normalized preview capture, quality presets, and explicit anchor-image selection when callers provide multiple references to a single-image-conditioned runtime
  • Hunyuan3D 2.1 batch generation for text-to-shape and image-to-shape runs, with compatibility-aware micro-batching on standard-memory GPUs and honest warm-sequential fallback when low-VRAM or mmgp staging is required
  • Hunyuan3D 2.1 mesh topology control for geometry-only OBJ output, including a documented PyMeshLab quad-dominant post-process on top of Tencent’s released triangle mesh generation path
  • a Hunyuan3D 2.1 style fine-tuning pipeline that prepares Tencent-compatible HunyuanDiT LoRA datasets, drives the official native DeepSpeed trainer, and emits reusable adapter manifests for local text-to-shape runs
  • an SF3D loader that registers Stability AI’s gated stable-fast-3d weights, resolves consumer-GPU fp16 and native bf16 load plans, and documents the compiled UV unwrap / texture baking runtime surface needed for later sub-second image-to-3D execution
  • a SPAR3D loader that registers Stability AI’s gated stable-point-aware-3d weights, resolves standard vs low-VRAM execution plans, and exposes the editable 512-point xyzrgb intermediate representation used between point diffusion and final mesh extraction
  • a Sparc3D loader for operator-supplied private pretrained bundles, with local-filesystem or private Hugging Face mirror sources, bundle-manifest validation, and explicit 1024³ readiness checks instead of pretending public checkpoints exist
  • a Sparc3D image-to-3D pipeline that executes an operator-supplied Python entrypoint from the imported private bundle, scales between 256³, 512³, and 1024³ resolution plans for quality-vs-speed tradeoffs, and returns provider-compatible GLB/OBJ/PLY assets
  • strict Sparc3D watertight-manifold validation that independently inspects OBJ and ASCII PLY topology, rejects meshes with holes or non-manifold edges in surfaceMode: 'watertight', and records print-ready diagnostics for 3D printing workflows
  • explicit Sparc3D open-surface analysis that preserves thin sheets and disconnected shells as first-class outputs, requires topology sidecars for GLB exports, and records boundary/component diagnostics instead of forcing everything into closed solids
  • a deterministic Sparc3D geometric-fidelity benchmark that compares the local Sparc3D stack against local TRELLIS.2 and Hunyuan3D 2.1 baselines on watertight, open-surface, thin-structure, and printable reconstruction cases
  • a MeshAnything V2 topology-cleanup handoff for Sparc3D that retopologizes the operator mesh through the released main.py CLI, preserves OBJ analysis sidecars for validation, and rechecks watertight/open-surface guarantees after cleanup instead of trusting the post-process blindly
  • a Kaolin FlexiCubes mesh-extraction path that can remesh finalized geometry from TRELLIS text/image/multi-image, Hunyuan3D 2.1 text/image/batch shape runs, TripoSG image/scribble/multi-view, and Sparc3D, while explicitly rejecting texture-first SF3D and SPAR3D requests where remeshing would discard baked materials
  • automatic MeshAnything V2 post-processing across TRELLIS text/image/multi-image, Hunyuan3D 2.1 text/image/batch shape runs, Sparc3D cleanup handoffs, and the TripoSG family whenever callers ask for topology controls those native runtimes do not honestly expose on their own
  • a TripoSG 1.5B loader that registers the official VAST-AI/TripoSG rectified-flow weights plus the required briaai/RMBG-1.4 background-removal dependency, preserves the upstream 8GB CUDA floor, and models a more realistic ~12GB operating envelope for end-to-end local runs
  • a runnable TripoSG image-to-3D pipeline that follows the released RMBG foreground-isolation and TripoSGPipeline sampling path, supports step / guidance / seed / face-count control, and returns geometry-only GLB, OBJ, or PLY assets
  • a runnable TripoSG-scribble sketch-to-3D pipeline that follows the released CFG-distilled prompt+scribble conditioning path, exposes scribble/prompt confidence controls, and returns rapid-prototyping geometry assets without enabling the flash decoder that upstream flags for boundary artifacts
  • a TripoSG multi-view consistency mode that honestly layers on top of the published single-image pipeline by batching per-view reconstructions and reranking them with geometric medoid scoring instead of claiming a non-public fused multi-view checkpoint
  • a TripoSG vs TripoSR comparison harness that runs the local TripoSG surface against the repo’s legacy Tripo baseline and returns an explicit validation report with verdict and confidence instead of hand-wavy quality claims
  • a runnable SPAR3D single-image pipeline that follows the released image-to-point-cloud-to-mesh path, exports the intermediate points.ply artifact alongside the textured GLB, and supports seed plus guidance control for the point diffusion stage
  • a runnable SPAR3D point-cloud editing pipeline that accepts generated or uploaded ply point clouds, applies translation / scale / recolor edits to the released 512-point xyzrgb representation, and remeshes the edited cloud back into a textured GLB
  • a persistent SPAR3D preview loop that keeps the released runtime loaded, reuses the prepared image and current point cloud across iterations, and regenerates interactive preview meshes from edited point clouds without paying cold-start model load cost on every edit
  • a runnable SF3D single-image pipeline that follows Stability AI’s released background-removal, foreground-cropping, run_image(...), and textured-GLB export path while surfacing precision downgrades and textured completion assets through the local provider contract
  • a speed-optimized SF3D batch image-to-3d pipeline that uses the released list-image run_image(...) path, compatibility-aware micro-batching, and VRAM-capped throughput heuristics to target 100+ images/minute on fast-path settings
  • a quality-vs-speed router that automatically keeps latency-sensitive image requests on SF3D, promotes fidelity-sensitive requests onto TRELLIS.2, and composes TRELLIS geometry with the released GLB/GLTF export stage so the quality route still returns delivery-ready assets
  • an SF3D UV unwrap export path that derives a UV layout artifact directly from the official Stability AI unwrapped mesh and returns it as a provider texture asset alongside the textured GLB
  • an SF3D illumination-disentangled texture export that surfaces the released albedo texture directly from the model’s material decoder, alongside normal, roughness, and metallic outputs needed for downstream PBR workflows

Subsequent Phase 71 tasks build on this package for containerization, GPU orchestration, model-specific runtime integrations, and cost tracking.

TRELLIS.2 Loader#

Local3DTrellis2ModelLoader is the first model-specific runtime integration in the package. It wraps the existing Hugging Face-backed Local3DModelRegistry and bakes in the official Microsoft upstream layout:

  • microsoft/TRELLIS.2-4B for the main image-to-3D and texturing checkpoints
  • microsoft/TRELLIS-image-large for the sparse-structure decoder still referenced by pipeline.json
  • runtime-side dependencies on facebook/dinov3-vitl16-pretrain-lvd1689m and briaai/RMBG-2.0

Example:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const trellis = createLocal3DTrellis2ModelLoader({ registry });
const plan = await trellis.resolveLoadPlan({
  precision: 'fp16',
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 24_576,
  },
});

The resolved plan includes:

  • the installed main and sparse-structure checkpoint locations
  • pipeline metadata from pipeline.json and texturing_pipeline.json
  • per-component native precision and runtime precision
  • automatic bf16 validation against GPU compute capability
  • flash-attn vs xformers attention backend selection
  • low-VRAM operating mode defaults and estimated runtime VRAM requirements

The default runtime profile is bf16 with an estimated 16GB VRAM requirement. Requesting fp16 switches the loader into low-VRAM mode and targets an 8GB operating profile by downcasting the bf16 flow checkpoints at load time while keeping the native fp16 decoder components unchanged.

Hunyuan3D 2.1 Shape Loader#

The package now also exposes Local3DHunyuan3D21ModelLoader for Tencent’s official Hunyuan3D 2.1 shape-generation runtime:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const hunyuan = createLocal3DHunyuan3D21ModelLoader({ registry });
const plan = await hunyuan.resolveLoadPlan({
  allowMemoryDowngrade: true,
  enableFlashVdm: true,
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 12_288,
  },
});

This loader is grounded in Tencent’s published Hunyuan3D 2.1 sources and model card:

  • the Hugging Face repo tencent/Hunyuan3D-2.1 publishes the shape DiT subfolder, optional ShapeVAE files, and the separate PBR paint bundle
  • the official runtime entrypoint is Hunyuan3DDiTFlowMatchingPipeline.from_pretrained(...)
  • Tencent’s README states a 10GB VRAM target for shape generation, while the upstream runtime also exposes low_vram_mode and CPU offload hooks

The resolved load plan includes:

  • selective download of only hunyuan3d-dit-v2-1/config.yaml and hunyuan3d-dit-v2-1/model.fp16.ckpt so the loader does not pull the 21GB paint bundle just to run shape inference
  • offline runtime hints for from_pretrained, including the local model path, subfolder, config path, checkpoint path, and the upstream conditioner->model->vae CPU offload order
  • three memory profiles: standard (10GB), low-vram (8GB with enable_model_cpu_offload), and mmgp (6GB staged profile coordinated by Local3DHunyuan3D21MmgpCoordinator)
  • auxiliary artifact metadata for the published ShapeVAE files so later fine-tuning and advanced VAE-specific workflows can reuse the same registry install metadata

Hunyuan3D 2.1 Text-to-Shape Pipeline#

The package now also exposes a runnable local text-to-shape pipeline on top of the Hunyuan loader:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21TextToShapePipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3D21TextToShapePipeline({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
  allowMemoryDowngrade: true,
});

const result = await pipeline.run({
  prompt: 'weathered bronze astrolabe',
  negativePrompt: 'busy background',
  meshTopology: 'quad-dominant',
  styleAdapterPath:
    '/models/hunyuan-style-loras/porcelain-style/adapter-manifest.json',
  styleAdapterScale: 0.8,
  quality: 'high',
  seed: 7,
});

This matches Tencent’s published text workflow:

  • the Hunyuan3D 2.1 gradio app enables text-to-3D by first generating a reference image from Tencent-Hunyuan/HunyuanDiT-v1.1-Diffusers-Distilled
  • the generated image then goes through optional background removal before Hunyuan3DDiTFlowMatchingPipeline.from_pretrained(...) reconstructs the shape
  • the Diffusers HunyuanDiT pipeline supports direct negative_prompt, num_inference_steps, and guidance_scale controls, which Oshun maps from the text request surface

At runtime the Node pipeline:

  • downloads the distilled HunyuanDiT Diffusers bundle plus the previously defined Hunyuan3D 2.1 shape checkpoint into Local3DModelRegistry
  • resolves the shape loader first so low-VRAM and 6GB mmgp fallbacks carry into the text flow automatically
  • invokes a package-local Python runner that emits staged progress for text-to-image loading, image generation, background removal, shape sampling, mesh extraction, and final asset writes
  • can fuse Tencent-native HunyuanDiT style LoRA adapters into the Diffusers text-to-image stage by pointing styleAdapterPath at either a checkpoint directory containing adapter_model.safetensors or the manifest emitted by the fine-tuning pipeline below
  • defaults quad-dominant requests to obj and runs PyMeshLab meshing_tri_to_quad_dominant as an explicit post-process, because Tencent’s released runtime still emits triangle meshes
  • routes targetPolyCount requests through an automatic MeshAnything V2 post-process after shape extraction so face-budget control is enforced instead of being treated as metadata-only intent
  • returns both the generated mesh and the reference PNG used for reconstruction, so callers can persist the exact image that conditioned the final shape

Hunyuan3D 2.1 Style LoRA Fine-Tuning#

Oshun now also packages a native HunyuanDiT LoRA fine-tuning pipeline for project-specific asset styles. This task is grounded in Tencent’s official HunyuanDiT LoRA workflow rather than an ad-hoc Diffusers training loop:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21FineTuningPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3D21FineTuningPipeline({
  registry,
  hyditTrainingRoot: '/opt/HunyuanDiT',
});

const result = await pipeline.run({
  adapterName: 'Porcelain Style',
  triggerPhrase: 'porcelain style',
  examples: trainingExamples,
  validationPrompts: ['garden pavilion', 'lotus flower'],
  rank: 64,
  maxTrainingSteps: 2_000,
});

The fine-tuning pipeline:

  • downloads Tencent’s native Tencent-Hunyuan/HunyuanDiT-v1.1 t2i/model checkpoint layout through Local3DModelRegistry so the official trainer sees the same pytorch_model_ema.pt and pytorch_model_module.pt structure it expects
  • writes a HunyuanDiT-compatible CSV caption file, converts it into Arrow shards via Tencent’s hydit/data_loader/csv2arrow.py, and emits the matching YAML index consumed by hydit/train_deepspeed.py
  • launches Tencent’s official DeepSpeed LoRA trainer with the published v1.1 style-conditioning recipe (--use-style-cond, --size-cond 1024 1024, --beta-end 0.03) instead of pretending the local Diffusers-distilled checkpoint can be trained natively
  • packages the latest checkpoint into an adapter-manifest.json file whose checkpointPath can be fed straight back into Local3DHunyuan3D21TextToShapePipeline through styleAdapterPath

Hunyuan3D 2.1 Image-to-Shape Pipeline#

The package now also exposes a direct image-conditioned shape pipeline:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21ImageToShapePipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3D21ImageToShapePipeline({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
  allowMemoryDowngrade: true,
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  meshTopology: 'quad-dominant',
  quality: 'high',
  seed: 7,
});

This follows the released Hunyuan3D 2.1 image-conditioned runtime directly:

  • it resolves the same Hunyuan3D 2.1 shape loader introduced in 71.1.3.1
  • it uses Tencent’s Hunyuan3DDiTFlowMatchingPipeline.from_pretrained(...) image path, with the same optional CPU-offload and flashvdm hooks
  • it can post-process the generated triangle mesh into a quad-dominant obj using PyMeshLab when callers request preserved face topology for downstream DCC workflows
  • it now also routes targetPolyCount requests through an automatic MeshAnything V2 remesh pass so face-count control works for image-conditioned runs and mixed-format outputs instead of remaining advisory
  • it emits the background-normalized preview image used for reconstruction so the exact conditioning input is preserved alongside the mesh

As of March 26, 2026, the published hunyuan3d-dit-v2-1/config.yaml still targets SingleImageEncoder plus ImageProcessorV2, not Tencent’s separate multi-view preprocessor. Oshun therefore handles multi-image requests by selecting one anchor image deterministically (first-image or largest-image) and records that decision in result metadata instead of pretending the released 2.1 checkpoint performs native multi-view fusion.

Hunyuan3D 2.1 Batch Generation#

The text-to-shape and image-to-shape pipelines now also expose runBatch(...) for throughput-oriented asset generation:

ts
const textBatch = await textPipeline.runBatch({
  targetBatchSize: 4,
  requests: [
    { prompt: 'bronze raven', outputFormat: 'glb', seed: 11 },
    { prompt: 'marble lion', meshTopology: 'quad-dominant', seed: 12 },
  ],
});

const imageBatch = await imagePipeline.runBatch({
  targetBatchSize: 4,
  requests: [
    { images: [frontImage], outputFormat: 'glb' },
    { images: [sideImage], quality: 'high' },
  ],
});

This batching path is grounded in Tencent’s released runtime rather than a fake queue-only abstraction:

  • the text batch runner feeds prompt lists through the Diffusers HunyuanDiTPipeline and then passes the resulting image list into Hunyuan3DDiTFlowMatchingPipeline
  • the image batch runner feeds a list of preprocessed conditioning images directly into Hunyuan3DDiTFlowMatchingPipeline, which Tencent’s published implementation already supports
  • Oshun groups requests by the inference settings that truly affect tensor compatibility, keeps one Python process and one loaded runtime per compatible cohort, and only splits batches when settings like guidance, octree resolution, style LoRA, or memory mode differ
  • standard-memory execution uses a conservative honest cap of 2 assets per micro-batch on 24GB-class GPUs and 4 on 48GB-class GPUs; low-VRAM and mmgp execution intentionally falls back to warm-sequential processing because CPU offload and staged residency erase the benefit of resident micro-batches
  • per-item output formats, preview PNGs, and optional quad-dominant PyMeshLab post-processing are still preserved after the shared batched inference pass
  • per-item targetPolyCount requests are preserved too, with automatic MeshAnything V2 post-processing applied only to the specific batch items that asked for face-budget control

The Hunyuan3D-Paint texture stack remains a single-mesh runtime upstream, so these batch APIs currently optimize geometry throughput first and leave texturing as an explicitly separate stage.

Hunyuan3D-Paint Loader#

The package now also exposes a dedicated loader for the published Hunyuan3D-Paint texture-generation stack:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3DPaintModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = createLocal3DHunyuan3DPaintModelLoader({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
});

const plan = await loader.resolveLoadPlan({
  maxSelectedViews: 6,
  resolution: 2048,
  allowMemoryDowngrade: true,
});

This loader is grounded in Tencent’s published paint runtime and separates the different moving parts explicitly:

  • it models Tencent’s documented baseline of roughly 21GB VRAM for 6 views at a 2048-map output profile, while exposing explicit 512, 1024, 2048, and 4096 texture-map targets through a runtime resolution plan
  • it selectively downloads only the hunyuan3d-paintpbr-v2-1 checkpoint files plus the required local facebook/dinov2-giant dependency through Local3DModelRegistry
  • it surfaces the additional runtime prerequisites Tencent documents but does not host in the model repo, including the RealESRGAN_x4plus.pth checkpoint, the custom rasterizer build, and the mesh-painter compile script
  • because Tencent does not publish a dedicated low-VRAM paint switch, Oshun’s low-vram and mmgp modes are exposed as explicit staged-runtime plans with warnings instead of being misrepresented as native upstream toggles
  • runtime hints include the paint config path, custom diffusers pipeline path, DINO path, RealESRGAN path, renderer entrypoints, component residency strategy, the selected view/bake/downsample profile for the requested texture size, and the albedo/MR material outputs expected by the released stack

Hunyuan3D-Paint PBR Texture Synthesis#

The package now also exposes a full offline texture-synthesis pipeline for the published Hunyuan3D-Paint runtime:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3DPaintTextureSynthesisPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3DPaintTextureSynthesisPipeline({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
  allowMemoryDowngrade: true,
});

const result = await pipeline.run({
  mesh: inputMeshBuffer,
  meshFormat: 'glb',
  prompt: 'weathered brass with dark patina in recessed areas',
  outputFormat: 'glb',
  textureResolution: 1024,
});

This pipeline follows Tencent’s released texture stack closely while fixing the gaps required for self-hosted production use:

  • it keeps the released hunyuan3d-paintpbr-v2-1 diffusion model fully offline by loading the already-registered local checkpoint directory instead of calling upstream snapshot_download(...) at inference time
  • it supports both image-guided and text-guided texture synthesis; when only a text prompt is supplied, Oshun uses a neutral white style canvas as the required image-conditioning anchor and records that decision in warnings and metadata
  • it stages multiview diffusion, RealESRGAN enhancement, and UV baking as separate phases so the low-vram loader profile can offload components between steps instead of keeping the whole Tencent stack resident at once
  • it exposes true texture-map size control for 512, 1024, 2048, and 4096 outputs; lower sizes use an oversampled bake plus deterministic downsampling, while 4096 switches into a native 4K bake path
  • it auto-downloads Tencent’s required RealESRGAN_x4plus.pth checkpoint into hy3dpaint/ckpt/ if the runtime checkout is missing it
  • it emits the textured mesh plus standalone albedo, metallic, roughness, metallicRoughness, and geometry-derived normal maps through the standard provider completion contract
  • GLB output uses Tencent’s own published PBR conversion utility so the embedded material matches the exported standalone texture maps

Hunyuan3D 2.1 Combined Shape+Texture Pipeline#

The package now also exposes an end-to-end orchestration pipeline that chains the released Hunyuan 2.1 shape stage into the local Hunyuan3D-Paint texture stage:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21ShapeAndTexturePipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3D21ShapeAndTexturePipeline({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
  allowMemoryDowngrade: true,
});

const result = await pipeline.run({
  prompt: 'weathered bronze astrolabe',
  outputFormat: 'glb',
  styleAdapterPath:
    '/models/hunyuan-style-loras/porcelain-style/adapter-manifest.json',
});

This combined pipeline is grounded in Tencent’s published end-to-end demo flow while staying explicit about the runtime model:

  • it supports both text-driven and image-driven entrypoints by delegating to the already-isolated Oshun shape pipelines and then feeding the exact emitted preview image into the paint stage as the texture-conditioning anchor
  • it reports both Tencent’s published co-resident combined budget (LOCAL_3D_HUNYUAN3D21_COMBINED_STANDARD_VRAM_MB, 29GB class) and Oshun’s actual sequential peak budget so schedulers can reason about the official full stack versus the implemented staged execution path
  • mmgp mode resolves to the documented 6GB-class sequential path by combining the existing shape and paint MMGP profiles, while standard mode preserves the published full-stack envelope in metadata
  • final provider completion metadata includes both the shape-stage and texture-stage metadata snapshots so downstream systems can audit the entire generation chain instead of only the last stage
  • text-mode combined runs pass styleAdapterPath and styleAdapterScale through to the HunyuanDiT reference-image stage before texturing, so a custom style LoRA still influences the geometry bootstrap even though the texture bake remains triangle-based
  • the textured pipeline stays triangle-based by design: quad-dominant requests are rejected here because the released Hunyuan3D-Paint UV baking path re-triangulates meshes during texturing, so Oshun only exposes preserved quad output on the standalone geometry-only shape pipelines

Hunyuan3D 2.1 MMGP Coordinator#

Local3DHunyuan3D21MmgpCoordinator is the consumer-GPU orchestration layer for the Hunyuan 6GB path. It sits above the existing shape and paint pipelines and uses the shared Local3DGpuMemoryManager to keep only one heavy stage resident at a time while tracing the exact stage sequence that ran:

ts
import {
  createLocal3DGpuDeviceConfig,
  createLocal3DHunyuan3D21MmgpCoordinator,
  createLocal3DHunyuan3D21ShapeAndTexturePipeline,
  Local3DModelRegistry,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  modelsRoot: '/models',
  cachePolicy: { maxDiskUsageGb: 400, minFreeDiskGb: 50 },
});

const pipeline = createLocal3DHunyuan3D21ShapeAndTexturePipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d',
});

const coordinator = createLocal3DHunyuan3D21MmgpCoordinator({
  gpuDevices: [
    createLocal3DGpuDeviceConfig(0, {
      name: 'RTX 3060',
      vramBudgetMb: 6_144,
    }),
  ],
});

const execution = await coordinator.runShapeAndTexture(pipeline, {
  prompt: 'weathered bronze astrolabe',
  memoryMode: 'mmgp',
});

console.log(execution.plan.stages.map((stage) => stage.stageId));
console.log(execution.trace.peakReservedMb);

The coordinator exposes planning helpers for standalone text-to-shape, image-to-shape, texture synthesis, and combined shape+texture runs. For 6GB cards it automatically caps the default GPU reserve so the published MMGP envelope can use the whole device instead of inheriting the server-wide 2GB reserve intended for multi-tenant deployments.

Hunyuan3D 2.1 Blender Addon Integration#

The package also ships a Blender-side integration surface for direct viewport generation. Oshun’s implementation follows Tencent’s official addon direction of embedding generation into the sidebar, but targets the existing local provider API instead of a one-off /generate route and uses job polling plus auto-import instead of a single blocking request.

ts
import { createLocal3DHunyuan3D21BlenderAddonIntegration } from '@isis/3d-inference-local';

const integration = createLocal3DHunyuan3D21BlenderAddonIntegration({
  apiBaseUrl: 'http://127.0.0.1:4413',
});

const descriptor = await integration.describeInstall();
await integration.copyAddonScript('/tmp/hunyuan3d-blender-addon');

console.log(descriptor.scriptPath);
console.log(descriptor.installSteps);

The bundled addon:

  • installs as a single hunyuan3d_2_1_blender_addon.py file
  • targets /v1/3d/capabilities, /v1/3d/text-to-3d/jobs, and /v1/3d/image-to-3d/jobs
  • submits preferredModelKey: 'hunyuan3d-2.1' so the local server can route the request into the Hunyuan stack
  • exposes prompt, negative prompt, image reference, quality, output format, texturing, texture resolution, and background-removal controls directly in the sidebar
  • polls job state in the background, supports cancellation, downloads the final model asset, and imports glb, gltf, obj, or ply output back into a dedicated Blender collection
  • can auto-select and frame the imported result so the finished mesh appears in the viewport with minimal operator work

Hunyuan3D 2.1 Multiview Reference Generation#

The package now also exposes a bridge pipeline that turns a single reference image into canonical front/left/back/right view images before later reconstruction stages:

ts
import {
  Local3DModelRegistry,
  createLocal3DHunyuan3D21MultiviewImageGenerationPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DHunyuan3D21MultiviewImageGenerationPipeline({
  registry,
  runtimeRoot: '/opt/Hunyuan3D-2.1',
  allowMemoryDowngrade: true,
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  quality: 'high',
  viewOrder: ['front', 'left', 'back', 'right'],
});

This pipeline is intentionally grounded in Tencent’s published 2.1 sources rather than inventing a fake native novel-view checkpoint:

  • as of March 26, 2026, Tencent’s released 2.1 stack still ships single-image shape reconstruction plus the separate Hunyuan3D-Paint multiview diffusion module, not a standalone single-image front/left/back/right generator
  • Oshun therefore bootstraps a coarse shape mesh with the released Hunyuan3DDiTFlowMatchingPipeline, renders normal and position control maps for the requested camera views, and then runs the published hunyuan3d-paintpbr-v2-1 diffusion bundle to synthesize canonical reference images
  • the pipeline downloads the exact paint checkpoint files plus the local facebook/dinov2-giant dependency through Local3DModelRegistry, keeps HF_HUB_OFFLINE=1 for the runner, returns the synthesized views as completion.textures[] with reference-view-* types, and preserves the background-normalized input preview plus the bootstrap mesh path for later debugging

Because this flow uses Tencent’s published paint runtime, the runtime root still needs a built Hunyuan3D 2.1 checkout with the renderer dependencies available under hy3dpaint/.

TRELLIS Text Pipeline#

The package now also ships a text-to-3D mesh pipeline for the official Microsoft TRELLIS text-conditioned checkpoints:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellisTextTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellisTextTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS',
});

const result = await pipeline.run({
  prompt: 'weathered bronze astrolabe',
  outputFormat: 'obj',
  seed: 7,
});

This pipeline is built from the current official upstream Microsoft sources:

  • microsoft/TRELLIS-text-large for the text-conditioned flow checkpoints
  • microsoft/TRELLIS-image-large for the shared decoders required by the text pipeline
  • the original microsoft/TRELLIS Python runtime, which currently contains the official text-conditioned pipeline implementation

This is an inference from the current upstream repos and model cards: the released TRELLIS.2 repo is image-conditioned today, while Microsoft’s text-conditioned weights remain on the original TRELLIS runtime.

At runtime the Node pipeline:

  • downloads the official checkpoint bundles into Local3DModelRegistry
  • rewrites the upstream pipeline.json dependency references into a fully local bundle so inference can run offline from the downloaded weights
  • executes the official Python pipeline via a package-local runner script
  • emits progress stages for runtime load, text encoding, sparse-structure sampling, SLat sampling, mesh extraction, and final mesh write-out
  • automatically hands the extracted mesh into MeshAnything V2 when callers ask for meshTopology or targetPolyCount controls beyond the native TRELLIS extractor surface

For this task the pipeline intentionally focuses on raw mesh extraction, so the supported output formats are obj and ply. Textured/PBR exports and GLB/GLTF packaging are implemented in later TRELLIS tasks.

TRELLIS.2 Image Pipeline#

For single-image reconstruction, the package now exposes a TRELLIS.2 image pipeline on top of the official trellis2 runtime:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2ImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellis2ImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS.2',
});

const result = await pipeline.run({
  images: [imageBuffer],
  outputFormat: 'obj',
  removeBackground: true,
  pipelineType: '1024_cascade',
});

At runtime the Node wrapper:

  • downloads microsoft/TRELLIS.2-4B plus the shared sparse-structure decoder from microsoft/TRELLIS-image-large
  • rewrites the upstream pipeline.json dependency reference into a local bundle so inference runs from registry-managed weights instead of remote downloads
  • executes only the geometry branch of the official TRELLIS.2 image pipeline for this task, producing raw mesh output in obj or ply
  • emits progress stages for image preprocessing, image feature extraction, sparse-structure sampling, shape SLat sampling, mesh extraction, and final write-out
  • optionally accepts meshExtractionResolution in 64-step increments between 64 and 512 so callers can trade extraction fidelity against decode cost

This task intentionally stops at raw mesh reconstruction. Multi-image conditioning, PBR extraction, Gaussian splatting output, and GLB/GLTF export are implemented in later TRELLIS subtasks.

TRELLIS.2 Multi-Image Pipeline#

For higher-fidelity reconstruction from multiple references, the package now also exposes a dedicated TRELLIS.2 multi-image pipeline:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2MultiImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellis2MultiImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS.2',
});

const result = await pipeline.run({
  images: [frontView, leftView, rightView, backView],
  outputFormat: 'obj',
  pipelineType: '1024_cascade',
  conditioningMode: 'concat',
});

At runtime the Node wrapper:

  • reuses the same patched local microsoft/TRELLIS.2-4B bundle as the single-image pipeline, so all checkpoint downloads stay registry-managed
  • accepts 2-6 input views, preprocesses each one independently, and passes them into a dedicated multi-image runner
  • fuses multiple image-conditioning feature sets into one TRELLIS conditioning sequence, with concat as the default strategy and mean as a lower-memory alternative
  • selects up to 4 conditioning views when more than 4 inputs are supplied, matching the current upstream training regime while still accepting 5-6 references from callers
  • supports the same optional meshExtractionResolution override, allowing lower-cost 64³-512³ extraction grids without changing the underlying TRELLIS pipeline type
  • automatically runs MeshAnything V2 after reconstruction when callers request meshTopology or targetPolyCount, so multi-image runs get the same honest topology-control surface as the single-image path
  • emits selected-view metadata so downstream systems know which input views were actually used for conditioning

This is based on the current upstream TRELLIS.2 training sources: the official dataset code samples up to 4 conditioning views, while community multi-image forks demonstrate multi-view inference on top of the released TRELLIS.2 image weights. Oshun therefore accepts 2-6 inputs, then selects and fuses the strongest subset for the official checkpoint bundle.

TRELLIS.2 PBR Material Extraction#

The package now also exposes a dedicated material extraction pipeline built on top of the official texturing_pipeline.json runtime:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2PbrMaterialExtractionPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellis2PbrMaterialExtractionPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS.2',
});

const result = await pipeline.run({
  mesh,
  meshFormat: 'obj',
  image: referenceImage,
  texturingResolution: 1024,
  textureSize: 2048,
});

This pipeline:

  • loads the official TRELLIS.2 texturing runtime via texturing_pipeline.json
  • encodes the input mesh back into shape SLat, samples texture SLat from the reference image, decodes the PBR voxel field, and rasterizes UV textures
  • returns typed texture assets for albedo, metallic, roughness, alpha, and a combined metallicRoughness map ready for later GLTF/GLB packaging
  • reuses the official UV-unwrapping and rasterization path from Microsoft’s texturing pipeline so the extracted maps stay aligned with upstream behavior

TRELLIS.2 Gaussian Splatting#

The package now also exposes a TRELLIS.2 Gaussian splatting export pipeline for real-time viewing stacks that prefer splats over triangle meshes:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2GaussianSplattingPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellis2GaussianSplattingPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS.2',
});

const result = await pipeline.run({
  mesh,
  meshFormat: 'obj',
  image: referenceImage,
  texturingResolution: 1024,
  gaussianCount: 32768,
});

This pipeline:

  • reuses the official TRELLIS.2 texturing runtime to infer appearance from the reference image instead of inventing a separate local material model
  • samples the normalized TRELLIS.2 mesh surface, queries the decoded PBR voxel field at each sampled point, and fits anisotropic surface-aligned Gaussian splats
  • writes a GraphDECO-compatible binary .ply asset with SH DC color, opacity, scale, and quaternion rotation fields so existing Gaussian viewers can ingest it directly
  • returns the asset through the standard model-completion contract using the logical output format gaussian-splatting

This is an inference from the current upstream sources, not a direct Microsoft feature: official TRELLIS.2 ships mesh and texturing decoders today, while the older microsoft/TRELLIS runtime defines the canonical Gaussian PLY schema and viewer/export path. Oshun combines those two upstream pieces so TRELLIS.2 outputs can participate in real-time Gaussian workflows immediately.

TRELLIS.2 GLB/GLTF Export#

The package now also exposes a dedicated GLB/GLTF export pipeline for turning a TRELLIS.2 mesh plus reference image into a textured delivery asset:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2GltfExportPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTrellis2GltfExportPipeline({
  registry,
  runtimeRoot: '/opt/TRELLIS.2',
});

const result = await pipeline.run({
  mesh,
  meshFormat: 'obj',
  image: referenceImage,
  outputFormat: 'glb',
  texturingResolution: 1024,
  textureSize: 2048,
});

This pipeline:

  • reuses the official TRELLIS.2 texturing runtime end-to-end, including mesh preprocessing, texture latent sampling, PBR voxel decode, and textured mesh postprocessing
  • exports textured GLB directly through the same trimesh-based path shown in the official example_texturing.py
  • postprocesses GLTF export bundles into a single .gltf file by inlining emitted image and buffer resources as data URIs, so callers do not need sidecar texture files
  • returns the final model asset through the standard completion contract with model/gltf-binary or model/gltf+json content typing

This is partly an inference from upstream exporter behavior: the official TRELLIS.2 example demonstrates single-file GLB export directly, while single-file embedded GLTF is assembled locally by inlining the trimesh exported resources into one JSON payload.

TRELLIS.2 Quality Presets#

resolveLocal3DTrellis2QualityPreset exposes the preset table used by the single-image and multi-image TRELLIS.2 pipelines. Explicit request fields still win, but when callers provide only quality, Oshun now maps that preset onto a concrete TRELLIS.2 runtime configuration:

  • draft: 512 pipeline, 128³ mesh extraction, 8 sparse-structure steps, 8 shape-SLat steps
  • standard: 1024_cascade pipeline, 256³ extraction, official released 12-step samplers
  • high: direct 1024 pipeline, 384³ extraction, 14-step samplers
  • ultra: 1536_cascade pipeline, 512³ extraction, 18-step samplers, and a larger token budget

This mapping is grounded in the released TRELLIS.2 resolution tiers and model card speed table: Microsoft documents 512³, 1024³, and 1536³ inference targets with roughly ~3s, ~17s, and ~60s runtimes on H100 hardware.

TRELLIS.2 Prompt Adapter#

The released TRELLIS.2 checkpoints are image-conditioned, so prompt engineering in Oshun works by translating Isis prompt templates into guidance for the upstream image-generation step rather than pretending the runtime consumes free text directly:

ts
import { adaptLocal3DTrellis2TextRequestPrompt } from '@isis/3d-inference-local';

const adaptation = adaptLocal3DTrellis2TextRequestPrompt({
  prompt: 'weathered bronze astrolabe with engraved markings',
  category: 'prop',
  style: 'photorealistic, cinematic',
  quality: 'high',
  generateTextures: true,
  pbrTextures: true,
});

The adapter emits:

  • a referenceImagePrompt tuned for clean single-object renders
  • a sectioned structuredPrompt split into subject, style, camera, lighting, background, and detail guidance
  • extracted material/style/detail tags so callers can inspect or postprocess the adaptation before handing it to an upstream image model
  • negative-prompt adaptation helpers that emit negativeReferenceImagePrompt guidance for excluding unwanted features before TRELLIS.2 reconstruction

When callers pass prompt into the TRELLIS.2 single-image or multi-image mesh pipelines, Oshun now surfaces the derived promptAdaptation metadata in the result payload instead of silently discarding the text.

When callers pass negativePrompt, Oshun now surfaces a negativePromptAdaptation payload that translates exclusions such as busy background, extra limbs, or text watermark into upstream reference-image avoidance guidance. This is an inference from the current upstream runtime: the released TRELLIS.2 image checkpoints still use zero-image neg_cond, so the negative prompt needs to act before reconstruction rather than inside the model.

Style guidance is now normalized through resolveLocal3DTrellis2StyleConditioning. The built-in presets map Isis style values such as photorealistic, stylized, low-poly, and cartoon onto consistent reference-image tags, lighting cues, and detail guidance so the upstream image generation step stays aligned with the intended 3D art direction.

TRELLIS.2 Seed Control#

TRELLIS.2 requests now normalize seed handling through one shared resolver:

  • omitting seed uses deterministic default seed 1
  • passing an explicit integer seed replays that exact value
  • passing seed: -1 requests a randomized seed, but Oshun records the resolved seed in result metadata so the run can still be replayed later

The normalized seed and seedSource fields are now attached to the single-image, multi-image, PBR extraction, GLTF export, and Gaussian splatting completion metadata.

TRELLIS.2 Benchmark Suite#

The package now also ships a deterministic benchmark harness for tracking whether the self-hosted TRELLIS.2 stack is actually surpassing commercial baselines on the kinds of assets where Oshun cares about quality:

ts
import { runDefaultLocal3DTrellis2BenchmarkSuite } from '@isis/3d-inference-local';

const report = runDefaultLocal3DTrellis2BenchmarkSuite({
  generatedAt: '2026-03-26T00:00:00.000Z',
});

The benchmark report includes:

  • weighted per-case scoring across geometry fidelity, topology cleanliness, texture fidelity, prompt alignment, multi-view consistency, thin-structure preservation, and material realism
  • provider profiles for trellis2-local, meshy-6, rodin-gen-1.5, and tripo-v2.5, including first-party reference URLs that document the current commercial API surface used for calibration
  • case-level winner detection plus pairwise TRELLIS-vs-baseline win rates and average score margins
  • threshold gates so CI can fail when TRELLIS.2 regresses against any baseline

The default dataset covers five representative asset classes:

  • hard-surface product detail
  • thin-structure preservation
  • stylized character prompt fidelity
  • multi-view product consistency
  • organic photoreal asset quality

Each case is intentionally benchmarked with modality-specific weights instead of one flat score. Multi-image cases emphasize view consistency, thin-structure cases emphasize open-space preservation, and stylized cases emphasize prompt and surface fidelity.

TRELLIS.2 Update Automation#

Local3DTrellis2ModelUpdateAutomation adds an operations-oriented update loop on top of the existing loader and registry:

ts
import {
  Local3DModelRegistry,
  createLocal3DTrellis2ModelUpdateAutomation,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const automation = createLocal3DTrellis2ModelUpdateAutomation({
  registry,
});

const check = await automation.checkForUpdates();
const result = await automation.pullLatest();

This automation now:

  • checks Microsoft GitHub in priority order: latest release, latest tag, then branch-head commit when the repo has no formal releases or tags
  • uses conditional If-None-Match polling on the active detection endpoint so repeated checks can reuse the cached upstream snapshot without wasting GitHub API quota
  • persists update state under the registry manifests directory, including the last seen upstream snapshot, last applied snapshot, last pulled runtime archive, and last synced Hugging Face weight revisions
  • downloads the upstream GitHub source tarball into a local runtime-archive cache whenever a new upstream snapshot is applied
  • keeps Hugging Face weight sync explicit: the automation still downloads the current TRELLIS.2 and sparse-structure checkpoint revisions through Local3DModelRegistry, while warning that Microsoft does not currently expose a direct GitHub-release-to-Hugging-Face revision mapping

As of March 26, 2026, the live Microsoft TRELLIS.2 repository does not expose a GitHub releases/latest payload or published tags through the public GitHub API, so this automation currently tracks main branch head commits in practice and upgrades automatically if formal releases or tags appear later.

Progress Streaming#

Local3DGpuModelServer now exposes progress publication and SSE fan-out:

ts
const server = createLocal3DGpuModelServer();

server.publishProgress({
  jobId: 'job-123',
  status: 'running',
  progressPct: 42,
  stage: 'mesh-generation',
  modelKey: 'trellis-2',
  gpuId: 0,
});

Clients can subscribe on the configured binding.ssePath and optionally filter by jobId:

text
GET /v1/3d/events?jobId=job-123
Last-Event-ID: 5

For Meshy-style task streams, the server also accepts:

text
GET /v1/3d/events/job-123/stream

The stream emits event: message payloads shaped like Meshy's task progress updates, and terminal states include the richer task payload fields such as type, timestamps, result, and task_error.

REST API#

Local3DProviderRestApi plugs into Local3DGpuModelServer so the same local runtime can expose job-oriented HTTP endpoints:

ts
const providerApi = createLocal3DProviderRestApi({
  capabilities: createLocal3DProviderCapabilities([
    'text-to-3d',
    'image-to-3d',
  ]),
  onSubmit(submission, api) {
    api.updateTask(submission.id, {
      status: 'processing',
      progress: 10,
      stage: 'bootstrapping-runtime',
    });
  },
});

const server = createLocal3DGpuModelServer({
  providerApi,
});

Supported routes:

text
GET    /v1/3d/capabilities
GET    /v1/3d/costs/dashboard
GET    /v1/3d/quality/hunyuan-vs-meshy/dashboard
GET    /v1/3d/jobs
GET    /v1/3d/jobs/:jobId
DELETE /v1/3d/jobs/:jobId
GET    /v1/3d/jobs/:jobId/stream
GET    /v1/3d/jobs/:jobId/assets/:assetId
POST   /v1/3d/text-to-3d/jobs
POST   /v1/3d/image-to-3d/jobs

Text submissions reuse the shared Isis-compatible request shape. Image submissions accept base64 strings or data: URLs, which the controller decodes back into Buffer[] payloads for the runtime hooks. Completed tasks can expose in-memory model, thumbnail, and texture assets as downloadable URLs while still publishing the same lifecycle over SSE.

Container Deployment#

The package now includes a package-scoped GPU container build under docker/. The image compiles only the local inference runtime surface with TypeScript and runs it on top of an NVIDIA CUDA runtime image with Node 22:

text
docker build -f docker/Dockerfile -t oshun/isis-3d-inference-local:latest .
docker run --rm --gpus all -p 4413:4413 oshun/isis-3d-inference-local:latest

For compose-based GPU startup, use:

text
docker compose -f docker/docker-compose.gpu.yml up --build

Runtime behavior is driven by environment variables such as LOCAL_3D_CAPABILITIES, LOCAL_3D_GPU_DEVICES_JSON, and LOCAL_3D_RUNTIME_MODULE. The container bootstrap can dynamically load a runtime module from disk, so later model-specific integrations can be mounted into the same base image without replacing the HTTP, SSE, and health-check layers.

Kubernetes Helm Chart#

The package now ships with a GPU-oriented Helm chart under helm/ for cluster deployments:

text
helm lint helm/
helm upgrade --install isis-3d-local helm/ \
  --namespace isis-system \
  --create-namespace

The chart defaults to:

  • scheduling only onto GPU nodes with node-type=gpu and nvidia.com/gpu.present=true
  • requesting and limiting one nvidia.com/gpu device per pod
  • provisioning persistent volumes for model weights and Hugging Face / Torch caches
  • mounting /dev/shm as memory-backed storage for inference workloads
  • exposing the runtime over a ClusterIP service on port 4413

For shared storage or externally supplied runtime modules, override values such as persistence.models.existingClaim, persistence.cache.existingClaim, and runtimeModules.configMapName/runtimeModules.existingClaim.

Windows Portable Deployment#

createLocal3DHunyuan3D21WindowsPortableDeployment(...) assembles a self-contained Windows Hunyuan3D 2.1 bundle around:

  • a prepared @isis/3d-inference-local app dist
  • a portable Node runtime
  • a portable Python runtime with the Hunyuan dependencies already installed
  • a prepared Hunyuan3D 2.1 checkout
  • optional preseeded model and Hugging Face caches
ts
const deployment = createLocal3DHunyuan3D21WindowsPortableDeployment({
  identity: {
    bundleId: 'hunyuan3d-2-1-win-portable',
    label: 'Hunyuan3D 2.1 Portable',
    version: '0.1.0',
  },
  sources: {
    appDistDir: 'dist/libs/isis/3d-inference-local',
    nodeRuntimeDir: 'vendor/node-win-x64',
    pythonRuntimeDir: 'vendor/python-win-x64',
    hunyuanRuntimeDir: 'vendor/Hunyuan3D-2.1',
    modelCacheDir: 'vendor/model-cache',
  },
});

await deployment.assembleBundle('dist/windows/hunyuan3d-2-1-portable');

The generated bundle includes:

  • scripts/RUN-API.cmd and scripts/RUN-API.ps1 to start the Isis-compatible /v1/3d server with a generated Hunyuan runtime module
  • scripts/RUN-GRADIO.cmd and scripts/RUN-GRADIO.ps1 to launch the upstream Hunyuan Gradio app when gradio_app.py is present in the bundled runtime
  • scripts/DOCTOR.cmd and scripts/DOCTOR.ps1 to verify the portable runtime layout before launch
  • portable-manifest.json and PORTABLE-README.md documenting the assembled artifact, runtime settings, launchers, and upstream references

The bundle runtime is backed by createLocal3DHunyuan3D21RuntimeHooks(...), which routes local REST submissions into the text-to-shape, image-to-shape, or combined shape-plus-PBR pipelines depending on whether the request asks for textures. The launchers set the Windows-specific environment for model caches, the prepared Hunyuan runtime root, portable Python, and the generated runtime-module path so the package can boot without Docker or a system-wide Python/Node installation.

This implementation is informed by the official Tencent-Hunyuan/Hunyuan3D-2.1 Windows guidance published with the June 13, 2025 open-source release and by the existing YanWenKun/Hunyuan3D-2-WinPortable community pack, both reviewed on 2026-03-26.

SF3D Loader#

Local3DSf3DModelLoader covers Stability AI’s official Stable Fast 3D runtime layout while keeping the package honest about the consumer-GPU constraints and gated model access:

ts
import {
  Local3DModelRegistry,
  createLocal3DSf3DModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const sf3d = createLocal3DSf3DModelLoader({ registry });
const plan = await sf3d.resolveLoadPlan({
  precision: 'fp16',
  allowPrecisionDowngrade: true,
  gpuDevice: {
    id: 0,
    computeCapability: '8.6',
    vramBudgetMb: 8_192,
  },
});

The loader is grounded in Stability AI’s published Stable Fast 3D sources as reviewed on 2026-03-26:

  • the gated Hugging Face repository stabilityai/stable-fast-3d exposes config.yaml and model.safetensors
  • the official runtime entrypoint is SF3D.from_pretrained(...)
  • the released run.py example uses CUDA torch.autocast(..., dtype=torch.bfloat16)
  • the official requirements include native texture_baker and uv_unwrapper packages, plus pynanoinstantmeshes for remeshing
  • Stability AI documents Windows support as experimental and recommends Visual Studio 2022 for extension builds

The resolved load plan includes:

  • gated-model install metadata and an explicit warning that Hugging Face access must be approved before automated downloads can succeed in a real environment
  • two half-precision operating profiles: fp16 for the ~4GB minimum consumer profile and bf16 for the ~6GB recommended profile on Ampere-or-newer GPUs
  • a documented downgrade path from bf16 to fp16 when the GPU lacks native bf16 support or enough VRAM, using allowPrecisionDowngrade
  • runtime hints for SF3D.from_pretrained(...), CUDA autocast, CPU fallback via SF3D_USE_CPU, supported remesh modes, and the upstream --texture-resolution CLI flag
  • explicit dependency manifests for the official repo, pinned Python packages, and the compiled texture_baker / uv_unwrapper extensions needed by later UV and texture-output tasks

The fp16 plan is intentionally labeled as an inference from the official sources rather than a direct upstream default: Stability AI’s published example uses bf16 autocast on CUDA, but the Phase 71 requirement is a half-precision consumer-GPU loader that can operate in the 4GB-to-6GB range when later pipelines are layered on top.

SPAR3D Loader#

Local3DSpar3DModelLoader extends the Stability AI integration with the newer point-aware runtime and makes the editable point-cloud intermediate explicit in the local model contract:

ts
import {
  Local3DModelRegistry,
  createLocal3DSpar3DModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const spar3d = createLocal3DSpar3DModelLoader({ registry });
const plan = await spar3d.resolveLoadPlan({
  allowMemoryDowngrade: true,
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 8_192,
  },
});

The loader is grounded in Stability AI’s published SPAR3D sources and model tree as reviewed on 2026-03-26:

  • the gated Hugging Face repository stabilityai/stable-point-aware-3d exposes config.yaml and model.safetensors
  • the official runtime entrypoint is SPAR3D.from_pretrained(...)
  • the released run.py example uses CUDA torch.autocast(..., dtype=torch.bfloat16), SPAR3D_LOW_VRAM, and return_points=True
  • the official spar3d.system path first generates a denoised pc_cond point-cloud condition and then feeds that editable point cloud into the final mesh stage
  • the official Gradio demo allows users to export, edit, re-import, subsample, and pad the point cloud before final meshing

The resolved load plan includes:

  • a standard runtime profile at roughly 10.5GB VRAM and a published low-VRAM fallback at roughly 7GB, with automatic downgrade via allowMemoryDowngrade
  • explicit runtime hints for SPAR3D.from_pretrained(...), SPAR3D_LOW_VRAM, CPU fallback via SPAR3D_USE_CPU, and experimental Apple Silicon fallback via PYTORCH_ENABLE_MPS_FALLBACK
  • a typed pointCloudIntermediate contract describing the released xyzrgb representation: 512 points, 6 channels, pc_cond as the batch key, pointcloud as the run_image(...) argument, and normalize_pc_bbox as the normalization step before meshing
  • low-VRAM dynamic-module paging metadata that mirrors the released point-diffusion, main-mesh, and estimator groups unloaded and reloaded by the upstream runtime
  • dependency manifests for the official repo, CLIP / AlphaCLIP, optional remesh packages, optional Gradio point-cloud editor packages, and the compiled texture_baker / uv_unwrapper extensions

The plan also preserves one upstream ambiguity instead of hiding it: Stability AI’s current README mentions both ~10.5GB default VRAM and ~6GB single-image manual inference. Oshun treats the explicit low-VRAM section as authoritative for fallback planning, keeps the discrepancy in warnings, and leaves later SPAR3D inference tasks to validate the true end-to-end footprint under the selected output settings.

Sparc3D Loader#

Local3DSparc3DModelLoader adds an honest loader contract for Sparc3D even though the public project still does not ship downloadable runtime weights:

ts
import {
  Local3DModelRegistry,
  createLocal3DSparc3DModelLoader,
  createLocal3DSparc3DOperatorBundleSource,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const sparc3d = createLocal3DSparc3DModelLoader({
  registry,
  weightsSource: createLocal3DSparc3DOperatorBundleSource({
    rootDir: '/models/private/sparc3d',
  }),
});

const plan = await sparc3d.resolveLoadPlan({
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 24_576,
  },
});

This loader is grounded in the live public Sparc3D materials reviewed on 2026-03-26:

  • the project page still says code may be released only upon approval of Math Magic, so Oshun does not pretend an official inference runtime or weight tree is publicly downloadable yet
  • the public Hugging Face space is demo-facing and does not expose a checkpoint bundle that Local3DModelRegistry can download directly
  • the paper and project page still establish the product direction: sparse high-resolution 1024^3 geometry, Sparconv-VAE, and Sparcubes decoding

The resolved load plan therefore treats Sparc3D as a validated private bundle format instead of a fake public repo:

  • Local3DModelRegistry can import the pretrained bundle from a local filesystem path or from a private mirrored Hugging Face repository
  • the bundle root must contain bundle.manifest.json, and that manifest must declare the image-encoder, sparconv-vae, latent-diffusion, and sparcubes-decoder checkpoint entries
  • the manifest must also advertise 1024 in runtime.supportedVoxelResolutions, so later high-resolution image-to-3D stages are blocked early if the wrong checkpoint family is staged
  • runtime precision, VRAM planning, latency, batch size, supported surface modes, and Python package requirements are all read from the bundle manifest; unknown values stay at 0 with an explicit warning instead of being guessed

This keeps Sparc3D usable inside the local model-registry flow today while remaining accurate about the current upstream publication gap.

Sparc3D Image-to-3D Pipeline#

Local3DSparc3DImageTo3DPipeline turns that private-bundle contract into an actual runnable image-to-3d stage with explicit 256^3, 512^3, and 1024^3 resolution plans:

ts
import {
  Local3DModelRegistry,
  createLocal3DSparc3DImageTo3DPipeline,
  createLocal3DSparc3DOperatorBundleSource,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const sparc3d = createLocal3DSparc3DImageTo3DPipeline({
  registry,
  weightsSource: createLocal3DSparc3DOperatorBundleSource({
    rootDir: '/models/private/sparc3d',
  }),
});

const result = await sparc3d.run({
  images: [imageBuffer],
  outputFormat: 'glb',
  quality: 'standard',
  surfaceMode: 'watertight',
  seed: 42,
});

This pipeline is grounded in the same public Sparc3D sources rechecked on 2026-03-27:

  • the public GitHub README still exposes only the paper/project links and the 1024^3 positioning, not a published inference runtime
  • the project page still says code may be released only upon approval of Math Magic, so the executable path remains operator-supplied rather than upstream downloadable

The runnable stage therefore uses a generic but explicit operator contract:

  • Local3DModelRegistry imports the full private bundle, including checkpoint files and the Python module named by runtime.packageRoot
  • the bundled runner loads bundle.manifest.json, resolves runtime.entrypoint, imports that module from the staged bundle, and invokes the callable with the image path, output path, requested voxel resolution, surface mode, seed, bundle manifest path, and resolved component paths
  • single-image conditioning is enforced up front; multi-image requests fail before runtime startup because the current Sparc3D task is specifically the single-image path
  • quality presets now map onto actual voxel plans: draft -> 256^3, standard -> 512^3, and high/ultra -> 1024^3
  • explicit voxelResolution overrides are supported when the operator bundle advertises that resolution in runtime.supportedVoxelResolutions

The result surface matches the local provider conventions used elsewhere in this package: one model asset (glb, obj, or ply), an optional prepared-image thumbnail, and structured metadata carrying the bundle revision, selected voxel resolution, scaled VRAM estimate, selected surface mode, seed, output format, runtime warnings, and any operator-reported mesh statistics.

Watertight Manifold Enforcement#

surfaceMode: 'watertight' now does more than forward a hint into the private bundle runtime:

  • Oshun independently parses the returned mesh topology for obj and ASCII ply outputs and computes boundary-edge, non-manifold-edge, degenerate-face, and connected-component counts before accepting the result as print-ready
  • glb exports remain supported, but the operator bundle must also emit an OBJ or ASCII-PLY analysis sidecar through analysis_mesh_path, because the final GLB container is not treated as a trustworthy topology-analysis source inside this library
  • if the mesh is not closed and manifold, the request fails instead of quietly returning a supposedly watertight asset
  • completion metadata now records watertightMesh, repairActions, and printableSolid, which makes the 3D-printing contract explicit to downstream orchestration

This is intentionally strict. Sparc3D's public paper and project page position the method around high-fidelity 1024^3 watertight surfaces, so Oshun treats a failed watertight verification as a hard correctness error rather than a soft warning.

Open-Surface Support#

surfaceMode: 'open-surface' now has a distinct validation path aimed at thin surfaces and disconnected parts:

  • Oshun uses the same OBJ / ASCII-PLY topology analysis flow to inspect boundary edges, connected components, degenerate faces, and non-manifold edges for open-surface requests
  • boundary edges and multiple connected components are treated as valid signals that thin sheets or floating sub-meshes survived reconstruction, instead of being misclassified as failures
  • glb exports must still include an analysis sidecar so the library can inspect whether open boundaries or loose parts were preserved
  • completion metadata now records openSurfaceMesh, including preservesOpenBoundaries, hasDisconnectedComponents, and supportsThinStructures

Open-surface mode still rejects clearly broken geometry such as non-manifold edges or degenerate faces. The difference from watertight mode is that boundary loops and disconnected components are allowed and explicitly surfaced instead of being treated as defects.

Geometric Fidelity Benchmark#

runDefaultLocal3DSparc3DGeometricFidelityBenchmark() adds a deterministic geometry-only comparison surface for the local Sparc3D, TRELLIS.2, and Hunyuan3D 2.1 stacks:

ts
import { runDefaultLocal3DSparc3DGeometricFidelityBenchmark } from '@isis/3d-inference-local';

const report = runDefaultLocal3DSparc3DGeometricFidelityBenchmark();
console.log(report.summary);

The benchmark intentionally avoids texture-centric scoring and instead focuses on the geometric claims that make Sparc3D interesting inside Oshun:

  • geometryFidelity and surfaceDetailRetention weight silhouette accuracy and dense relief retention
  • watertightReliability measures printable-solid performance on closed-shape cases
  • openSurfacePreservation and thinStructurePreservation keep the open-surface and delicate-geometry paths first-class instead of forcing every provider into a watertight-only rubric
  • pairwise report rows expose Sparc3D's win rate and worst-case margin against the local TRELLIS.2 and Hunyuan3D 2.1 baselines

Like the existing TRELLIS and Hunyuan dashboards, this is a deterministic evaluation dataset checked into the package, which keeps CI coverage reliable and makes geometry-regression thresholds explicit without requiring live model runs during tests.

MeshAnything V2 Cleanup#

The package now also exposes a dedicated MeshAnything V2 model loader so the official Hugging Face weights and runtime caveats can be planned explicitly before any cleanup or remeshing workflow runs:

ts
import {
  Local3DModelRegistry,
  createLocal3DMeshAnythingV2ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });
const loader = createLocal3DMeshAnythingV2ModelLoader({ registry });

const plan = await loader.resolveLoadPlan();

console.log(plan.install.installDir);
console.log(plan.runtime.maxOutputFaces);
console.log(plan.warnings);

This loader is grounded in the public MeshAnything V2 release reviewed on 2026-03-27:

  • the canonical official Hugging Face repository is Yiwen-ntu/MeshAnythingV2 and the published weight set currently includes config.json, model.safetensors, and 350m.pth
  • Oshun models MeshAnything V2 as a remeshing family member even though the Hugging Face card uses image-to-3d, because the released runtime actually consumes dense meshes or point clouds through main.py
  • the published operational envelope is about 8GB VRAM and 45s per mesh on an A6000, while Oshun recommends extra headroom for Marching Cubes preprocessing and flash-attn allocation spikes
  • the upstream model class still bootstraps config from facebook/opt-350m, so fully offline deployments must pre-cache that config or patch the runtime to stay air-gapped
  • the Git checkout remains part of the runtime contract because the released entrypoint imports mesh_to_pc.py plus MeshAnything/miche/shapevae-256.yaml from the repository tree
  • outputs remain within the current upstream <=1600 face artist-mesh regime and should be treated as topology cleanup / controlled retopology rather than unlimited mesh refinement

Sparc3D can now hand its topology-analysis OBJ into the released MeshAnything V2 CLI as an explicit post-process step:

ts
import {
  Local3DModelRegistry,
  createLocal3DSparc3DImageTo3DPipeline,
  createLocal3DSparc3DOperatorBundleSource,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });

const sparc3d = createLocal3DSparc3DImageTo3DPipeline({
  registry,
  weightsSource: createLocal3DSparc3DOperatorBundleSource({
    rootDir: '/models/private/sparc3d',
  }),
  meshAnythingRuntimeRoot: '/models/runtime/MeshAnythingV2',
});

const result = await sparc3d.run({
  images: [imageBuffer],
  outputFormat: 'glb',
  surfaceMode: 'watertight',
  topologyCleanup: 'meshanything-v2',
  quality: 'high',
});

This integration is deliberately grounded in the current upstream MeshAnything V2 surface reviewed on 2026-03-27:

  • the official repository exposes main.py for mesh-inference rather than a stable Python library API, so Oshun drives that CLI through a dedicated runner
  • upstream recommends --mc preprocessing for non-Marching-Cubes meshes; Oshun enables it by default for watertight Sparc3D cleanup and disables it for surfaceMode: 'open-surface' so intended boundary loops are not sealed away
  • the current upstream model targets an artist-mesh envelope capped by the released <=1600 face regime, so cleanup is treated as topology improvement with an expected density reduction rather than a fidelity-preserving no-op

The returned metadata now includes topologyCleanup with before/after face and vertex counts, boundary / manifold deltas, Marching Cubes settings, and cleanup warnings. Oshun then reruns the same watertight or open-surface verification on the cleaned mesh before returning it, so a post-process cannot silently break the geometry contract established by the original Sparc3D request.

The same MeshAnything V2 runtime can now also be reused as an automatic post-process helper across the other generation families in this package:

  • TRELLIS text, TRELLIS.2 single-image, and TRELLIS.2 multi-image now automatically remesh when callers request meshTopology or targetPolyCount
  • Hunyuan3D 2.1 text/image single runs, both shape batch APIs, and the combined shape+texture orchestrator now enforce targetPolyCount through the same post-process helper
  • TripoSG image, scribble, and multi-view consistency runs automatically hand quad/topology-sensitive requests to MeshAnything V2 instead of leaving them as warning-only intent
  • Sparc3D automatically activates its MeshAnything cleanup handoff when callers ask for face-budget or topology controls, while surfacing that the dedicated animation-ready edge-flow pass currently lives on the MeshAnything remeshing API rather than Sparc3D's own request layer

FlexiCubes Mesh Extraction#

meshExtractionMethod: 'flexicubes' now requests a Kaolin-backed FlexiCubes remesh across the geometry-first generation families in this package:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });

const pipeline = createLocal3DTripoSGImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TripoSG',
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  outputFormat: 'obj',
  quality: 'high',
  meshExtractionMethod: 'flexicubes',
});

This feature is intentionally honest about the current upstream boundaries:

  • the official NVIDIA Kaolin surface reviewed on 2026-03-27 is kaolin.non_commercial.flexicubes.FlexiCubes, so Oshun now drives the real Kaolin extractor instead of pretending native model outputs were already FlexiCubes meshes
  • most released 3D runtimes used here return finalized meshes rather than caller-controlled signed-distance fields, so Oshun reconstructs a temporary scalar field from nearest-surface point-and-normal projection before invoking FlexiCubes
  • TRELLIS text/image/multi-image, Hunyuan3D 2.1 text/image/batch shape runs, TripoSG image/scribble/multi-view, and Sparc3D all expose this as a real post-extraction path with per-run flexiCubesPostProcess metadata
  • Sparc3D keeps an OBJ or ASCII-PLY analysis sidecar when FlexiCubes is used so open-surface / watertight validation and optional MeshAnything cleanup still inspect the remeshed geometry instead of stale native output
  • SF3D and SPAR3D now reject meshExtractionMethod: 'flexicubes' explicitly because those released Stability AI pipelines bake textured GLB assets from their native mesh path and Oshun cannot re-extract geometry there without discarding baked materials

FlexiCubes now also defaults to adaptive resolution on top of Kaolin's public regular voxel-grid API. Oshun keeps the real construct_voxel_grid(...) topology exposed by Kaolin, then applies a bounded axis warp derived from face-dihedral and open-boundary complexity histograms before reconstructing the scalar field. That gives flatter regions coarser cell spacing while redistributing density toward thin, high-curvature, and boundary-heavy areas without inventing an upstream octree API that Kaolin does not currently expose.

Grid selection is still preset-driven at the coarse level: draft maps to 128, mid-tier requests map to 192, and higher-detail plans map to 256 unless a family already exposes a stronger native resolution signal such as Sparc3D voxel resolution. Low-level callers can still set adaptivityMode: 'uniform' on the FlexiCubes postprocessor when they need a strictly uniform reconstruction grid for controlled experiments or comparisons.

The low-level FlexiCubes postprocessor also now exposes an optional differentiableRefinement stage for iterative mesh cleanup:

ts
import { createLocal3DFlexiCubesAutomaticPostProcessor } from '@isis/3d-inference-local';

const processor = createLocal3DFlexiCubesAutomaticPostProcessor({
  workDir: '/tmp/oshun-flexicubes',
});

const refined = await processor.run({
  sourceLabel: 'Sparc3D image-to-3d generation',
  mesh: inputMeshBuffer,
  meshFormat: 'obj',
  outputFormat: 'obj',
  gridResolution: 256,
  differentiableRefinement: {
    iterations: 48,
    cameraBatchSize: 4,
    trainResolution: 256,
  },
});

This path follows the official FlexiCubes optimization workflow rather than a made-up post-smoothing pass: Oshun initializes the FlexiCubes SDF from the input mesh, renders mask/depth supervision from that source mesh with nvdiffrast, and then optimizes FlexiCubes SDF values, voxel deformations, and per-cube weights before re-extracting the final surface. Because this is a real differentiable optimization loop, it is intentionally opt-in on the low-level API instead of silently running on every FlexiCubes request.

For regression tracking, the package now also exposes a deterministic FlexiCubes-vs-Marching-Cubes benchmark harness through runDefaultLocal3DFlexiCubesVsMarchingCubesBenchmark(...). The default dataset captures five equal-budget extraction cases across hard-surface, thin-structure, photogrammetry, organic, and optimization-oriented workloads, and scores both extractors on geometry fidelity, sharp feature retention, triangle quality, topology cleanliness, thin-structure preservation, and optimization readiness. That keeps the quality claim tied to an explicit, versioned benchmark surface instead of ad hoc eyeballing.

When CUDA is available, the FlexiCubes postprocessor now also uses Kaolin's mesh-ops stack directly for the heavy mesh-prep stages instead of relying only on trimesh and a CPU KD-tree. Surface sampling runs through kaolin.ops.mesh.sample_points, nearest-face normals come from kaolin.ops.mesh.face_normals, the scalar field is built with kaolin.metrics.trianglemesh.point_to_mesh_distance, and watertight sources use kaolin.ops.mesh.check_sign for sign classification. Open meshes still get the accelerated Kaolin distance path, but switch sign inference to a nearest-face-normal heuristic, and full CPU fallback is recorded explicitly in the returned kaolinMeshOps report metadata and warning stream.

MeshAnything V2 Remeshing Pipeline#

The package now also exposes a dedicated remeshing pipeline for direct dense mesh cleanup:

ts
import {
  Local3DModelRegistry,
  createLocal3DMeshAnythingV2RemeshingPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });

const pipeline = createLocal3DMeshAnythingV2RemeshingPipeline({
  registry,
  runtimeRoot: '/models/runtime/MeshAnythingV2',
});

const result = await pipeline.run({
  mesh: inputDenseMeshBuffer,
  meshFormat: 'obj',
  meshTopology: 'quad-dominant',
  animationReadyTopology: true,
  animationRigProfile: 'humanoid',
  edgeFlowStrength: 'balanced',
});

This pipeline is grounded in the current upstream MeshAnything V2 runtime, but it closes the practical gaps needed for local production use:

  • it resolves the MeshAnything V2 Hugging Face weights through the local model registry and passes the downloaded weight directory into the runtime instead of pretending a hard-coded remote model id is sufficient
  • the Python runner now drives the upstream mesh_to_pc.py plus MeshAnythingV2.from_pretrained(...) path directly, which keeps the released geometry logic while avoiding the earlier remote-only CLI handoff
  • Oshun patches the upstream facebook/opt-350m config bootstrap to use the local config.json when a local model directory is supplied, so the remesh path can run without a hidden extra network fetch
  • meshTopology: 'quad-dominant' is implemented as a real two-stage cleanup: MeshAnything V2 first produces the artist-mesh triangle result, then Oshun runs a quad-dominant conversion pass and preserves the result as OBJ
  • animationReadyTopology: true adds an explicit edge-flow optimization stage between the artist mesh and the quad conversion pass; Oshun uses articulation band heuristics plus documented PyMeshLab remeshing/edge-flip filters so likely neck, shoulder, elbow, hip, knee, and ankle regions can be biased toward cleaner rigging loops for generic-biped, humanoid, and quadruped assets
  • face-count control now snaps requests onto the supported 200, 400, 800, and 1600 artist-mesh budgets and applies that simplification before any optional quad conversion, so later topology stages work from the requested density envelope instead of a fixed upstream output
  • the pipeline exports both the normalized preprocessed input mesh and the MeshAnything artist-mesh OBJ sidecar so topology analysis and later quality scoring can compare before/after structure instead of only the final file
  • returned metadata includes normalized input, artist-mesh, and final-output topology summaries, plus the Marching Cubes setting, seed, and warnings that affected the cleanup path
  • animation-ready runs also return an edgeFlowOptimization report with the optimized articulation bands, quad coverage, valence-4 ratio, edge-length consistency, and a heuristic closed-loop score for the final OBJ topology
  • every remesh run now also returns a topologyQualityComparison report that scores the normalized pre-remesh mesh against the post-remesh topology on boundary-edge ratio, non-manifold/degenerate penalties, connected-component count, face-regularity, valence variance, and target-topology preference so Oshun can quantify whether the remesh materially improved topology quality

MeshAnything V2 LOD Chain Generation#

The package now also exposes a dedicated MeshAnything V2 LOD-chain generator for game-oriented asset packaging:

ts
import {
  Local3DModelRegistry,
  createLocal3DMeshAnythingV2LodChainGenerator,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });

const generator = createLocal3DMeshAnythingV2LodChainGenerator({
  registry,
  runtimeRoot: '/models/runtime/MeshAnythingV2',
});

const result = await generator.run({
  mesh: inputDenseMeshBuffer,
  meshFormat: 'obj',
  outputFormat: 'glb',
});

This chain generator is grounded in the same upstream MeshAnything V2 limits and the published MSFT_lod ordering model:

  • Oshun emits LOD0 as the highest-detail MeshAnything cleanup pass, then derives LOD1 through LOD4 from the supported 1600, 800, 400, and 200 face budgets
  • the returned manifest includes descending screen-coverage thresholds aligned with the high-to-low ordering used by the Khronos MSFT_lod extension, so downstream engines can wire the exported files into their own LOD switching policy without reverse-engineering the chain order
  • each LOD preserves the same topology preferences as the source remesh request, including quad-dominant conversion and the optional animation-ready edge-flow pass, instead of silently dropping those constraints on lower-detail levels
  • Oshun currently exports the chain as separate OBJ/PLY/GLB assets plus a JSON manifest rather than merging them into a single glTF container, which keeps the output honest and compatible with non-glTF game build pipelines

MeshAnything V2 Batch Remeshing#

The package now also exposes a dedicated MeshAnything V2 bulk-cleanup pipeline for asset-library remediation:

ts
import {
  Local3DModelRegistry,
  createLocal3DMeshAnythingV2BatchRemeshingPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({ rootDir: '/models/isis-3d' });

const pipeline = createLocal3DMeshAnythingV2BatchRemeshingPipeline({
  registry,
  runtimeRoot: '/models/runtime/MeshAnythingV2',
});

const result = await pipeline.runBatch(
  [
    {
      mesh: firstDenseMeshBuffer,
      meshFormat: 'obj',
      outputFormat: 'glb',
      meshTopology: 'triangle',
    },
    {
      mesh: secondDenseMeshBuffer,
      meshFormat: 'obj',
      animationReadyTopology: true,
      animationRigProfile: 'humanoid',
      targetFaceCount: 800,
    },
  ],
  { requestedBatchSize: 2 }
);

This batch pipeline is grounded in the released MeshAnything V2 runtime and the upstream folder-input execution model, but it fills in the production gaps for real library cleanup:

  • the public main.py --input_dir ... path is treated as the honest upstream batching surface, and Oshun converts that into manifest-driven CUDA micro-batches instead of pretending the original single-file CLI was already a full provider-ready batch API
  • compatible jobs share one MeshAnything V2 model load and one batched forward pass while still preserving per-item output formats, face-count simplification, normalized-input sidecars, and artist-mesh analysis OBJs
  • compatibility grouping is explicit: Marching Cubes enablement/resolution, the sampling flag, and stochastic sampling seeds must match before assets are placed into the same micro-batch
  • Oshun keeps the VRAM envelope honest by defaulting to micro-batch size 2 when no GPU budget is supplied, allowing 2 on typical 24GB cards, and only exposing 4 on 48GB+ plans; 256-resolution Marching Cubes and stochastic sampling stay capped at 2
  • post-remesh topology work still happens per asset, so quad-dominant conversion, animation-ready edge-flow optimization, topology-quality scoring, and provider-compatible completion payloads survive the batch path instead of being silently dropped for throughput
  • returned batch metadata now reports compatibility-group count, micro-batch count, effective execution mode, and measured throughput in items per minute so operators can reason about whether the cleanup job actually delivered library-scale gains

SPAR3D Image-to-3D Pipeline#

Local3DSpar3DImageTo3DPipeline turns the released SPAR3D point-aware runtime into a provider-compatible single-image workflow and keeps the intermediate point cloud visible instead of hiding it behind the final mesh:

ts
import {
  Local3DModelRegistry,
  createLocal3DSpar3DImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSpar3DImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/stable-point-aware-3d',
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  quality: 'standard',
  guidanceScale: 3,
  seed: 1234,
});

This pipeline is grounded in the same official SPAR3D sources reviewed on 2026-03-26, but it intentionally follows the lower-level Gradio flow instead of the one-line CLI helper so the editable point cloud remains available:

  • image preprocessing still uses the released transparent-background foreground-removal path plus foreground_crop(...)
  • point diffusion is run explicitly through forward_pdiff_cond(...) and sampler.sample_batch_progressive(...) so guidanceScale and seed are honored for the intermediate pc_cond
  • the intermediate 512-point xyzrgb cloud is exported as points.ply alongside the final textured mesh.glb
  • final meshing still uses the released generate_mesh(...) path with optional none / triangle / quad remeshing and texture baking

The returned provider completion includes:

  • a primary textured glb asset
  • a secondary ply model asset containing the intermediate point cloud
  • a thumbnail of the processed RGBA input used for reconstruction
  • metadata for point-cloud generation time, mesh generation time, point count, selected memory mode, guidance scale, seed, and the emitted runtime keys

standard quality maps to the released 1024 texture-resolution default and stays closest to the published fast path. high and ultra remain available, but the pipeline warns when those settings move the run away from the fastest advertised SPAR3D profile.

SPAR3D Point-Cloud Editing#

Local3DSpar3DPointCloudEditingPipeline exposes the same editable intermediate that Stability AI’s published Gradio demo uses and turns it into a provider-safe local workflow:

ts
import {
  Local3DModelRegistry,
  createLocal3DSpar3DPointCloudEditingPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSpar3DPointCloudEditingPipeline({
  registry,
  runtimeRoot: '/opt/stable-point-aware-3d',
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  quality: 'standard',
  editOperations: [
    { kind: 'translate', pointIndices: [0, 1, 2], delta: [0.05, 0.0, -0.03] },
    { kind: 'recolor', pointIndices: [0, 1, 2], color: [255, 180, 32] },
  ],
});

This pipeline is grounded in the same official SPAR3D sources reviewed on 2026-03-26, including the published Gradio demo and runtime internals:

  • when pointCloudData is omitted, Oshun follows the released image-to-point diffusion path to generate the editable pc_cond
  • when pointCloudData is supplied, the pipeline accepts an uploaded ply point cloud, normalizes it to SPAR3D’s 512-point conditioning size through padding or subsampling, and bypasses point-diffusion-only controls such as guidanceScale and seed
  • edit operations are applied sequentially to the xyzrgb cloud before generate_mesh(...) reconstructs the final textured mesh
  • the supported edit primitives match the concrete editing affordances the official demo advertises directly: point translation, point scaling, and point recoloring
  • the edited point cloud is exported as ply alongside the final textured glb, so callers can round-trip into external tools such as MeshLab or Blender for more advanced edits and then re-run meshing locally

The returned provider completion includes:

  • a primary textured glb asset generated from the edited point cloud
  • a secondary ply asset containing the final edited conditioning cloud
  • a thumbnail of the processed RGBA input image used for conditioning
  • metadata for point-cloud source, edit operation count, applied operation kinds, edit time, mesh time, point count, and the selected memory mode

standard quality stays aligned with the released 1024-texture fast path. Higher quality tiers remain available for better texture fidelity, but the pipeline preserves the same warnings when those settings move the run away from the advertised low-latency SPAR3D profile.

SPAR3D Preview Loop#

Local3DSpar3DPreviewLoop turns SPAR3D’s published “edit point cloud, then re-run meshing” demo pattern into a real warmed local session instead of a cold-start wrapper:

ts
import {
  Local3DModelRegistry,
  createLocal3DSpar3DPreviewLoop,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loop = createLocal3DSpar3DPreviewLoop({
  registry,
  runtimeRoot: '/opt/stable-point-aware-3d',
});

const { session } = await loop.start({
  images: [inputImageBuffer],
  quality: 'high',
  meshTopology: 'quad-dominant',
  targetVertexCount: 3200,
});

const fastPreview = await session.preview({
  editOperations: [
    { kind: 'translate', pointIndices: [0, 1, 2], delta: [0.04, 0, -0.02] },
  ],
});

const finalMesh = await session.finalize();
await session.close();

This preview loop is grounded in the same official SPAR3D sources reviewed on 2026-03-26, especially the published Gradio demo’s “Re-run with point cloud” flow and the released generate_mesh(...) runtime internals:

  • the Python worker loads SPAR3D.from_pretrained(...) once, preprocesses the conditioning image once, and keeps the current 512-point pc_cond resident between renders
  • session.preview(...) defaults to the real fast-path settings for interactive feedback: 512 texture resolution and remesh: 'none'
  • session.finalize(...) uses the higher-quality defaults captured at session start, so callers can iterate quickly and then produce the final mesh without reloading the model
  • session.reset() restores the original generated or uploaded point cloud so a preview loop can recover from bad edits without re-running image conditioning
  • uploaded replacement point clouds still go through the same 512-point normalization that the official demo applies before meshing

Each preview/final render returns:

  • a textured glb mesh for the current edited point cloud
  • a ply export of the current conditioning cloud after any new edits
  • the prepared conditioning thumbnail
  • per-render metadata for edit counts, render mode, mesh timings, point-cloud source, and texture/remesh settings

The preview loop is intentionally conservative about what counts as “interactive”: raising texture resolution above 512 or enabling remeshing during session.preview(...) is allowed, but surfaced as a warning because it moves the session away from the released fast edit path.

SF3D Image-to-3D Pipeline#

Local3DSf3DImageTo3DPipeline wraps the official Stable Fast 3D single-image runtime into the Oshun provider surface and keeps the released preprocessing and export path intact:

ts
import {
  Local3DModelRegistry,
  createLocal3DSf3DImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSf3DImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/stable-fast-3d',
  defaultPrecision: 'fp16',
  allowPrecisionDowngrade: true,
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  quality: 'standard',
  meshTopology: 'quad-dominant',
  targetVertexCount: 12000,
});

This implementation is based on Stability AI’s released code paths reviewed on 2026-03-26:

  • run.py removes the background, resizes the foreground crop, runs SF3D.from_pretrained(...).run_image(...), and exports mesh.glb
  • gradio_app.py uses the same foreground_ratio preprocessing and drives the direct generate_mesh(...) path with a textured GLB preview
  • sf3d/system.py handles UV unwrap, texture baking, and PBR material assembly during mesh generation rather than as a post-export step

The pipeline surface intentionally reflects those upstream behaviors:

  • only a single image is accepted for this task because the released SF3D runtime is single-image conditioned
  • output is restricted to textured glb so the provider completion preserves the official textured-mesh artifact instead of discarding materials
  • quality maps onto texture-atlas size: draft=512, standard=1024, high=2048, ultra=4096
  • meshTopology maps onto the official remesh options: triangle -> triangle, quad-dominant -> quad, unset -> none
  • the completion payload includes both the textured GLB and the processed input preview used for reconstruction

At runtime the Node wrapper:

  • resolves the previously introduced SF3D load plan so gated-model warnings, precision downgrades, and VRAM estimates stay aligned with the loader
  • writes the incoming image buffer to disk, launches the bundled Python runner, and streams JSON progress for runtime load, preprocessing, mesh generation, texture bake, and final export
  • preserves the upstream remove_background and resize_foreground flow, surfaces peak memory and generation duration metadata from the runner, and returns a provider completion asset with model/gltf-binary content

SF3D Batch Image-to-3D Pipeline#

Local3DSf3DBatchImageTo3DPipeline exposes the real SF3D batched inference path from Stability AI’s published CLI instead of simulating batching at the queue layer:

ts
import {
  Local3DModelRegistry,
  createLocal3DSf3DBatchImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSf3DBatchImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/stable-fast-3d',
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 24_576,
  },
});

const result = await pipeline.runBatch(
  [{ images: [chairImage] }, { images: [lampImage] }, { images: [shoeImage] }],
  { requestedBatchSize: 8 }
);

This batch pipeline is grounded in the same official SF3D sources reviewed on 2026-03-26:

  • Stability AI’s published run.py already accepts --batch_size and passes a list of prepared images into model.run_image([...])
  • the bundled batch runner preserves the official preprocessing sequence for every item: RGBA load, optional rembg, and resize_foreground(...)
  • compatible requests are grouped by precision, background-removal settings, foreground ratio, texture resolution, remesh mode, and target vertex count so each micro-batch maps cleanly onto one native SF3D invocation
  • the micro-batch size is capped by a conservative VRAM heuristic derived from the already-documented 4GB fp16 and 6GB bf16 runtime envelopes
  • remeshing requests intentionally fall back to warm-sequential execution because the per-mesh CPU remesh stage is not the released high-throughput path

The batch result includes:

  • one provider-compatible Local3DSf3DImageTo3DResult per input request
  • batch metadata for compatibility groups, micro-batch count, effective batch size, total duration, and measured items-per-minute throughput
  • a targetThroughputAchieved flag checked against the Phase 71 goal of 100 images per minute

draft and standard quality with remesh: 'none' are the intended throughput settings. Higher texture resolutions or remeshing remain available, but the pipeline warns when those controls move a run away from the fastest official SF3D path.

Quality-vs-Speed Routing#

Local3DQualityVsSpeedRouter promotes the first automatic model-selection layer inside the local 3D stack. It is intentionally narrow and honest: SF3D remains the default speed path for single-image preview-style requests, while TRELLIS.2 takes over when the request is clearly quality-driven or needs a delivery format that the current SF3D surface does not expose.

ts
import {
  Local3DModelRegistry,
  createLocal3DQualityVsSpeedRouter,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const router = createLocal3DQualityVsSpeedRouter({
  registry,
  sf3dRuntimeRoot: '/opt/stable-fast-3d',
  trellisRuntimeRoot: '/opt/TRELLIS.2',
});

const speedResult = await router.runImage({
  images: [thumbnailImage],
  quality: 'draft',
});

const qualityResult = await router.runImage({
  images: [heroReference],
  quality: 'ultra',
  outputFormat: 'glb',
});

The router is grounded in the same upstream implementations reviewed on 2026-03-26:

  • Stability AI’s released SF3D stack is the fast single-image textured GLB path with published sub-second latency and native batched run_image([...])
  • Microsoft’s released TRELLIS.2 runtime is the higher-fidelity path with documented 512³ / 1024³ / 1536³ quality tiers and a first-party GLB export flow via o_voxel.postprocess.to_glb(...)
  • Oshun’s TRELLIS route composes those existing local TRELLIS geometry and GLB export stages so a quality-routed request does not regress from “delivery asset” back to “raw mesh intermediate”

Current routing rules:

  • single-image draft and latency-sensitive standard requests default to SF3D
  • high / ultra quality, prompt-guided image requests, and non-GLB delivery requests route to TRELLIS.2
  • multi-image requests stay on TRELLIS.2 because SF3D remains single-image
  • explicit quad-dominant topology requests can stay on SF3D when that topology control matters more than TRELLIS fidelity heuristics

The router exposes selectImageRoute(...), runImage(...), runBatch(...), and runText(...). Mixed batches are supported: speed-route items are collapsed into the SF3D batch pipeline when available, while quality-route outliers run through TRELLIS sequentially with per-item routing metadata preserved.

Intelligent Model Routing#

Local3DIntelligentModelRouter is the broader model-selection layer for Phase 71. It keeps the existing quality-vs-speed router intact, but expands the decision space across the local 3D families already implemented in this package:

  • organic and character-leaning prompts favor Hunyuan3D 2.1
  • hard-surface, architectural, and multi-image requests favor TRELLIS.2
  • speed-critical single-image preview jobs favor SF3D
  • high-resolution, printable, or thin-structure geometry jobs favor Sparc3D
  • sketch and doodle-flavored image prompts favor TripoSG
ts
import {
  Local3DModelRegistry,
  createLocal3DIntelligentModelRouter,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const router = createLocal3DIntelligentModelRouter({
  registry,
  sf3dRuntimeRoot: '/opt/stable-fast-3d',
  trellisRuntimeRoot: '/opt/TRELLIS.2',
  hunyuanRuntimeRoot: '/opt/hunyuan3d',
  triposgRuntimeRoot: '/opt/TripoSG',
});

const decision = router.selectImageRoute({
  images: [referenceImage],
  prompt: 'high-resolution printable ornate statue with fine filigree',
  quality: 'ultra',
  outputFormat: 'obj',
});

const result = await router.runImage({
  images: [referenceImage],
  prompt: 'hard-surface mech cockpit with industrial panels',
  outputFormat: 'glb',
});

The routing heuristics were aligned against the released upstream capability surfaces reviewed on 2026-03-27 and then mapped onto Oshun’s existing local pipelines:

  • Hunyuan3D 2.1 is the strongest local path for organic, production-style asset generation and direct GLB delivery
  • TRELLIS.2 remains the broadest geometry-plus-texturing path for hard-surface and multi-image reconstruction
  • SF3D is still the fastest released single-image textured preview route
  • Sparc3D is the geometry-first route when watertight, printable, or high-resolution reconstruction matters most
  • TripoSG is the best fit for sketch-like inputs and stylized reference images

Every routed result preserves a structured routing block in provider completion metadata with the selected family, selected model key, execution mode, candidate scores, prompt-analysis evidence, and human-readable reasons. TRELLIS image routes also preserve the existing geometry-plus-GLB/GLTF export composition so a routed request can still come back as a delivery-ready asset instead of an intermediate mesh only.

Quality-Aware Model Routing#

Local3DQualityAwareModelRouter sits on top of the intelligent router and executes multiple compatible candidates before selecting a winner. The current implementation is benchmark-backed rather than hand-wavy:

  • candidate families are shortlisted from the existing intelligent-router prompt analysis and compatibility scores
  • compatible candidates are executed in parallel or in bounded parallel cohorts depending on maxConcurrentCandidates
  • each successful candidate is scored from a weighted mix of benchmark prior, prompt-fit prior, artifact-quality signals, delivery fitness, latency efficiency, and warning penalties
  • benchmark priors come from the existing TRELLIS.2 benchmark suite, the Hunyuan3D 2.1 quality dashboard, and the Sparc3D geometric-fidelity benchmark; SF3D and TripoSG use explicit local heuristics because Phase 71 does not yet include equivalent comparative dashboards for them
  • failed candidates are preserved in the quality-routing report and excluded from ranking instead of aborting the whole request
ts
import {
  Local3DModelRegistry,
  createLocal3DQualityAwareModelRouter,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const router = createLocal3DQualityAwareModelRouter({
  registry,
  trellisRuntimeRoot: '/opt/TRELLIS.2',
  hunyuanRuntimeRoot: '/opt/hunyuan3d',
  triposgRuntimeRoot: '/opt/TripoSG',
  maxConcurrentCandidates: 2,
});

const result = await router.runImage({
  images: [referenceImage],
  prompt: 'printable cathedral ornament with fine filigree detail',
  outputFormat: 'obj',
  quality: 'ultra',
});

console.log(result.decision.selectedFamily);
console.log(
  result.candidates.map((candidate) => ({
    family: candidate.family,
    rank: candidate.rank,
    score: candidate.score?.totalScore,
  }))
);

Cost-Aware Model Routing#

Local3DCostAwareModelRouter keeps self-hosted generation as the default path and only falls back to configured cloud providers once the selected local route is genuinely blocked by GPU saturation or hard self-hosted capacity limits.

  • the router reuses the existing intelligent local-routing decision instead of inventing a second local selection stack
  • scheduler snapshots are inspected to determine whether the selected local model can run immediately, whether every eligible GPU is saturated, and how much queue delay the request is likely to inherit
  • queue-delay budgets are priority-aware by default (interactive, batch, background) and can be overridden per request
  • cloud providers are filtered by request compatibility before selection; exact output-format support, texture-resolution limits, multi-image support, PBR support, variation limits, and advanced mesh-control safety are all checked
  • when cloud fallback is needed, the router ranks compatible providers by known cloud price profile first and latency second
  • advanced local-only controls such as meshExtractionMethod, meshTopology, and quadDominantOptimization keep the request on the self-hosted path even under saturation rather than silently degrading them

The cloud-provider interface is intentionally structural, so existing provider instances that expose generateFromText(...), generateFromImage(...), and getCapabilities() can be passed in directly. That keeps @isis/3d-inference-local buildable without taking a hard compile-time dependency on the broader hosted-provider package surface.

ts
import { createLocal3DCostAwareModelRouter } from '@isis/3d-inference-local';

const router = createLocal3DCostAwareModelRouter({
  intelligentRouter,
  scheduler,
  queue,
  costDashboard,
  cloudProviders: [
    { id: 'meshy', provider: meshyProvider },
    { id: 'rodin', provider: rodinProvider },
  ],
});

const result = await router.runImage({
  images: [referenceImage],
  prompt: 'high-detail production prop',
  quality: 'high',
  outputFormat: 'glb',
  priority: 'interactive',
});

console.log(result.decision.selectedTarget);
console.log(result.decision.loadAssessment.estimatedQueueDelayMs);
console.log(result.completion.metadata?.costRouting);

Cloud-fallback executions are normalized back into the same provider-completion shape as local executions, persisted into the configured workDir, and tagged with a costRouting metadata block that preserves the selected target, selected local model key, selected cloud provider, queue/load evidence, and the estimated self-hosted versus cloud cost comparison.

The result surface keeps the winning routed output at the top level, preserves the full winning intelligent-route result under winner, and attaches a structured qualityRouting metadata block to provider completion metadata with candidate ranks, score margins, benchmark sources, and the scoring weights used for the final decision.

Generation A/B Testing#

Local3DGenerationABTestingFramework provides a deterministic experiment layer for Phase 71 model-routing work. It is purpose-built for comparing the three surfaces Oshun actually changes in this package:

  • model-version upgrades
  • prompt-strategy rewrites
  • post-processing pipeline swaps

The framework keeps the statistical surface explicit instead of hiding it behind "experiment" marketing language:

  • subjects are assigned to variants deterministically with sticky hashing
  • every experiment has an explicit control variant plus a declared primary metric
  • success-rate uses a two-sample z-test for proportions
  • quality-score, latency-ms, cost-usd, and gpu-utilization-pct use a Welch-style two-sample t-test
  • the snapshot reports confidence intervals, p-values, effect sizes, sample-size sufficiency, and a recommendation that stays conservative until the minimum sample target is reached
ts
import {
  createLocal3DGenerationABTestingFramework,
  createLocal3DModelVersionExperimentDefinition,
} from '@isis/3d-inference-local';

const experiments = createLocal3DGenerationABTestingFramework();

experiments.createExperiment(
  createLocal3DModelVersionExperimentDefinition({
    id: 'hunyuan-2-1-patch',
    name: 'Hunyuan 2.1 patch validation',
    primaryMetric: {
      id: 'quality-score',
    },
    secondaryMetrics: [
      {
        id: 'latency-ms',
      },
      {
        id: 'cost-usd',
      },
    ],
    minimumSampleSizePerVariant: 24,
    variants: [
      {
        id: 'control',
        label: 'Hunyuan 2.1.0',
        modelKey: 'hunyuan3d-2.1',
        modelVersion: '2.1.0',
        isControl: true,
      },
      {
        id: 'candidate',
        label: 'Hunyuan 2.1.1',
        modelKey: 'hunyuan3d-2.1',
        modelVersion: '2.1.1',
      },
    ],
  })
);

experiments.startExperiment('hunyuan-2-1-patch');

const assignment = experiments.assignVariant('hunyuan-2-1-patch', 'subject-42');

experiments.recordOutcome('hunyuan-2-1-patch', {
  subjectId: 'subject-42',
  variantId: assignment.variantId,
  success: true,
  qualityScore: 0.87,
  latencyMs: 18400,
  costUsd: 0.21,
});

const snapshot = experiments.getSnapshot('hunyuan-2-1-patch');
console.log(snapshot.summary.winningVariantId);
console.log(snapshot.primaryMetricComparisons[0]?.pValue);

Generation Analytics Dashboard#

Local3DGenerationAnalyticsDashboard is the operational dashboard surface for the same routing stack. It tracks the signals that matter when model routing is making production decisions instead of benchmark-only ones:

  • per-model success and failure rates
  • quality scores extracted from routed result metadata
  • average, p50, and p95 latency
  • GPU utilization from live sample capture or embedded task telemetry
  • self-hosted cost plus cloud-price comparison deltas
  • optional experiment summaries when the A/B framework is attached
ts
import {
  createLocal3DGenerationABTestingFramework,
  createLocal3DGenerationAnalyticsDashboard,
  createLocal3DProviderRestApi,
} from '@isis/3d-inference-local';

const experiments = createLocal3DGenerationABTestingFramework();
const analytics = createLocal3DGenerationAnalyticsDashboard({
  experimentFramework: experiments,
});

analytics.recordGpuUtilizationSample({
  gpuId: 0,
  utilizationPct: 72,
  memoryUtilizationPct: 58,
  activeModelKeys: ['hunyuan3d-2.1'],
});

const providerApi = createLocal3DProviderRestApi({
  capabilities,
  generationAnalyticsDashboard: analytics,
});

const snapshot = analytics.getSnapshot({
  cloudProfileIds: ['meshy-pro'],
});

console.log(snapshot.summary.successRate);
console.log(snapshot.modelBreakdown[0]?.averageLatencyMs);
console.log(snapshot.gpuBreakdown[0]?.latestUtilizationPct);
console.log(
  analytics.getGpuUtilizationSamples({
    gpuId: 0,
    since: new Date(Date.now() - 15 * 60_000),
  }).length
);

When the dashboard is attached to Local3DProviderRestApi, it is also exposed as a REST snapshot:

  • GET /v1/3d/analytics/generations/dashboard
  • filters: kind, status, modelKey, gpuId, limit, selfHostedProfileId, cloudProfileIds, and experimentStatus

GPU Utilization Monitor#

Local3DGpuUtilizationMonitor consumes those sampled GPU traces plus the multi-model scheduler snapshot and turns them into actionable cluster-hygiene alerts:

  • underutilized GPUs are flagged when a recent utilization window stays below a configurable threshold
  • idle resident model instances are flagged when they keep holding VRAM without active assignments for too long
  • active and resolved alerts keep timestamps and stable identifiers so external automation can forward only state changes
ts
import {
  Local3DGpuUtilizationMonitor,
  createLocal3DGenerationAnalyticsDashboard,
  createLocal3DMultiModelScheduler,
} from '@isis/3d-inference-local';

const analytics = createLocal3DGenerationAnalyticsDashboard();
const scheduler = createLocal3DMultiModelScheduler({
  gpuDevices,
  models,
  runtime,
});

const monitor = new Local3DGpuUtilizationMonitor({
  analytics,
  scheduler,
  gpuObservationWindowMs: 15 * 60_000,
  idleModelThresholdMs: 10 * 60_000,
  onAlertEvent: ({ transition, alert }) => {
    console.log(transition, alert.kind, alert.message);
  },
});

const cycle = await monitor.reconcile();
console.log(cycle.activeAlerts.map((alert) => alert.id));
console.log(cycle.idleModelObservations[0]?.idleDurationMs);

Model Sharding Across Multiple GPUs#

Local3DModelShardingManager adds tensor-parallel planning and reservation tracking for models that do not fit on a single GPU:

  • it estimates full-model VRAM with the same scheduling heuristic already used by the single-GPU scheduler
  • it chooses the smallest viable shard count that fits the cluster, while preferring homogeneous GPU sets over mixed hardware groups
  • it produces an explicit tensor-parallel plan with split dimension, collective ops, per-shard reservations, and load/unload hooks for the runtime layer
ts
import {
  Local3DModelShardingManager,
  type Local3DModelShardingRuntime,
} from '@isis/3d-inference-local';

const runtime: Local3DModelShardingRuntime = {
  async loadShardedModel(model, context) {
    console.log(model.key, context.plan.shardCount, context.plan.gpuIds);
  },
  async unloadShardedModel(model, context) {
    console.log('unload', model.key, context.reason);
  },
};

const manager = new Local3DModelShardingManager({
  gpuDevices: [
    {
      id: 0,
      name: 'RTX 4090',
      vramBudgetMb: 24_576,
      reserveMb: 0,
      modelAffinity: [],
    },
    {
      id: 1,
      name: 'RTX 4090',
      vramBudgetMb: 24_576,
      reserveMb: 0,
      modelAffinity: [],
    },
    {
      id: 2,
      name: 'RTX 4090',
      vramBudgetMb: 24_576,
      reserveMb: 0,
      modelAffinity: [],
    },
  ],
  models: [largeModelDescriptor],
  runtime,
});

const decision = await manager.schedule({
  modelKey: largeModelDescriptor.key,
  request: {
    prompt: 'polished brass astrolabe',
    quality: 'high',
  },
});

console.log(decision.plan.parallelismMode);
console.log(decision.plan.shardCount);
console.log(
  decision.shards.map((shard) => [shard.gpuId, shard.estimatedVramMb])
);

Model Quantization Planning#

Local3DModelQuantizationPlanner adds shared INT8 and INT4 planning across the self-hosted model families without forcing each loader to reinvent quantization rules:

  • it understands every built-in 3D family shipped by the package
  • it models calibration requirements, preserved modules, artifact outputs, and backend dependencies
  • it produces quantized runtime environment wiring plus VRAM and throughput estimates on top of an existing self-hosted load plan
ts
import {
  Local3DModelQuantizationPlanner,
  Local3DModelRegistry,
  Local3DTrellis2ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = new Local3DTrellis2ModelLoader({ registry });
const basePlan = await loader.resolveLoadPlan({
  precision: 'bf16',
  lowVram: false,
});

const quantizer = new Local3DModelQuantizationPlanner();
const int8Plan = quantizer.plan(basePlan, {
  precision: 'int8',
  calibrationSampleCount: 256,
});

console.log(int8Plan.estimatedRuntimeVramMb);
console.log(int8Plan.quantization.artifacts[0]?.absolutePath);
console.log(int8Plan.environment.LOCAL_3D_QUANTIZATION_BACKEND);

TensorRT Optimization Planning#

Local3DModelTensorRTPlanner adds a shared NVIDIA TensorRT planning surface for the self-hosted 3D runtimes instead of scattering ad hoc compile settings across individual model loaders:

  • it models family-specific dynamic-shape profiles for image, texturing, and remeshing paths
  • it plans persistent Torch-TensorRT program, engine-cache, timing-cache, and calibration-cache artifacts per GPU architecture
  • it exposes compile settings, environment wiring, VRAM reduction, and expected speedup on top of an existing load plan
ts
import {
  Local3DModelRegistry,
  Local3DModelTensorRTPlanner,
  Local3DTrellis2ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = new Local3DTrellis2ModelLoader({ registry });
const basePlan = await loader.resolveLoadPlan({
  precision: 'bf16',
});

const tensorrt = new Local3DModelTensorRTPlanner();
const optimizedPlan = tensorrt.plan(basePlan, {
  precision: 'fp16',
  capability: 'image-to-3d',
  targetGpu: {
    id: 0,
    name: 'NVIDIA L40S',
    computeCapability: '8.9',
    vramBudgetMb: 49_152,
  },
});

console.log(optimizedPlan.tensorrt.estimatedSpeedup);
console.log(optimizedPlan.tensorrt.compileSettings.engineCacheDir);
console.log(optimizedPlan.environment.LOCAL_3D_TENSORRT_PROGRAM_PATH);

ONNX Runtime Planning#

Local3DModelOnnxRuntimePlanner adds a cross-platform ONNX Runtime planning surface for the self-hosted 3D model families:

  • it maps each supported capability onto platform-valid execution providers for CUDA, DirectML, CoreML, and CPU fallback
  • it plans optimized ORT artifacts plus serialized session/provider profiles for deployment reuse
  • it exposes provider-aware precision fallback, graph optimization defaults, runtime dependencies, and environment wiring on top of an existing load plan
ts
import {
  Local3DModelOnnxRuntimePlanner,
  Local3DModelRegistry,
  Local3DSf3DModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = new Local3DSf3DModelLoader({ registry });
const basePlan = await loader.resolveLoadPlan({
  precision: 'fp16',
});

const onnx = new Local3DModelOnnxRuntimePlanner();
const optimizedPlan = onnx.plan(basePlan, {
  capability: 'image-to-3d',
  targetOs: 'windows',
  targetArch: 'x64',
});

console.log(optimizedPlan.onnxruntime.provider);
console.log(optimizedPlan.onnxruntime.sessionOptions.graphOptimizationLevel);
console.log(optimizedPlan.environment.LOCAL_3D_ONNX_OPTIMIZED_MODEL_PATH);

Compile Optimization Planning#

Local3DModelCompileOptimizationPlanner centralizes torch.compile planning for the PyTorch-native generation families and exposes the specialized stable-fast path for SF3D:

  • it limits compile planning to generation-oriented model families instead of pretending remeshing-only runtimes should be compiled the same way
  • it reuses existing runner wiring like Hunyuan3D 2.1’s --compile switch where the repo already exposes one
  • it models TorchInductor cache directories, stable-fast runtime dependencies, warmup costs, and expected steady-state latency gains on top of a load plan
ts
import {
  Local3DModelCompileOptimizationPlanner,
  Local3DModelRegistry,
  Local3DSf3DModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = new Local3DSf3DModelLoader({ registry });
const basePlan = await loader.resolveLoadPlan({
  precision: 'fp16',
});

const compiler = new Local3DModelCompileOptimizationPlanner();
const optimizedPlan = compiler.plan(basePlan, {
  backend: 'stable-fast',
  capability: 'image-to-3d',
});

console.log(optimizedPlan.compilation.estimatedSpeedup);
console.log(optimizedPlan.environment.LOCAL_3D_COMPILE_BACKEND);
console.log(optimizedPlan.runtimeDependencies.map((dep) => dep.dependencyId));

MeshAnything V2 Speculative Decoding#

Local3DMeshAnythingV2SpeculativeDecodingPlanner adds MeshAnything-specific draft/verify planning for the autoregressive retopology path:

  • it pairs the published MeshAnything V2 verifier with the same facebook/opt-350m bootstrap model family as a lightweight draft decoder
  • it estimates token budgets from the supported 200/400/800/1600-face buckets and emits draft-window, verification-batch, and acceptance-threshold settings
  • it exposes reusable artifact locations plus deterministic prefix-reconciliation helpers for verifier rollback handling
ts
import {
  Local3DMeshAnythingV2SpeculativeDecodingPlanner,
  Local3DModelRegistry,
  Local3DMeshAnythingV2ModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const loader = new Local3DMeshAnythingV2ModelLoader({ registry });
const basePlan = await loader.resolveLoadPlan();

const planner = new Local3DMeshAnythingV2SpeculativeDecodingPlanner();
const speculative = planner.plan(basePlan, {
  targetFaceCount: 1600,
});

console.log(speculative.speculativeDecoding.estimatedSpeedup);
console.log(speculative.environment.LOCAL_3D_SPECULATIVE_DRAFT_MODEL);
console.log(speculative.speculativeDecoding.artifacts[0]?.absolutePath);

Inference Result Caching#

Local3DInferenceResultCache caches completed local 3D inference outputs so repeated requests can short-circuit expensive generation work:

  • it hashes normalized request inputs together with model key, model version, optional family, and cache salt so prompt-identical runs only hit when the runtime identity also matches
  • it supports both text-to-3D and image-to-3D requests, hashing image bytes rather than Buffer identity for deterministic cache keys
  • it stores full completion payloads with TTL, byte-budget, and LRU eviction semantics so repeated results can be returned without rerunning inference
ts
import {
  Local3DInferenceResultCache,
  buildLocal3DInferenceCacheKey,
} from '@isis/3d-inference-local';

const cache = new Local3DInferenceResultCache({
  defaultTtlMs: 30 * 60 * 1000,
});

const cacheKey = buildLocal3DInferenceCacheKey({
  modelKey: 'trellis-2',
  modelVersion: '2.0.1',
  request: {
    prompt: 'ornate shield',
    quality: 'high',
  },
});

const hit = cache.get({
  modelKey: 'trellis-2',
  modelVersion: '2.0.1',
  request: {
    prompt: 'ornate shield',
    quality: 'high',
  },
});

console.log(cacheKey);
console.log(hit.hit);

Progressive Generation With Early Preview#

Local3DProgressiveGenerationCoordinator adds a shared two-stage orchestration surface for returning a draft preview quickly and then refining to the final asset:

  • it applies a conservative preview profile by default, using draft quality, glb output, a 400-face target, 512 texture resolution, and textures disabled so the preview path stays on the fast side of a 2-second target
  • it preserves the original request for the refinement pass, emits typed progress updates for preview preparation, preview readiness, refinement, and final completion, and records whether the preview met the requested deadline
  • it merges preview affordances such as previewUrl and thumbnails into the final completion payload so clients can keep showing the early artifact while higher-quality geometry and textures arrive
ts
import { Local3DProgressiveGenerationCoordinator } from '@isis/3d-inference-local';

const coordinator = new Local3DProgressiveGenerationCoordinator({
  previewRunner: async ({ request }) => ({
    previewUrl: `https://cdn.local/previews/${request.quality}.glb`,
  }),
  finalRunner: async () => ({
    models: [{ data: Buffer.from('final-glb'), format: 'glb' }],
  }),
});

const result = await coordinator.run(
  {
    jobId: 'job-42',
    request: {
      prompt: 'ornate bronze helmet',
      quality: 'high',
      textureResolution: 2048,
      generateTextures: true,
    },
  },
  {
    onProgress(update) {
      console.log(update.stage, update.progressPct);
    },
  }
);

console.log(result.firstPreviewLatencyMs);
console.log(result.mergedCompletion.previewUrl);
console.log(result.final.completion.models?.[0]?.format);

SF3D UV Unwrap Output#

Local3DSf3DUvUnwrapPipeline exposes the released SF3D UV unwrap as an explicit artifact instead of leaving it implicit inside the textured GLB:

ts
import {
  Local3DModelRegistry,
  createLocal3DSf3DUvUnwrapPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSf3DUvUnwrapPipeline({
  registry,
  runtimeRoot: '/opt/stable-fast-3d',
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  meshTopology: 'quad-dominant',
  textureResolution: 2048,
});

This task is grounded in the official Stability AI UV path reviewed on 2026-03-26:

  • sf3d/models/mesh.py imports the native uv_unwrapper package and calls Mesh.unwrap_uv(...)
  • the unwrap step duplicates seam vertices and stores per-vertex UVs on mesh.v_tex
  • sf3d/system.py then bakes textures against those UVs before constructing the textured trimesh export

The Oshun UV export keeps that official unwrap intact and adds explicit output artifacts on top:

  • the pipeline still returns the textured glb produced by the released SF3D runtime so downstream tools keep the same mesh/material representation
  • it also renders a uv-layout.png artifact directly from the official mesh UVs and exposes it as a provider texture asset with textureType=uv-layout
  • metadata includes UV vertex count, UV face count, UV bounds, and a UV area coverage estimate so later tasks can validate unwrap quality and atlas use

This is intentionally different from a generic downstream UV pass. The UV layout is derived from the exact Stability AI mesh that already passed through uv_unwrapper, which is the main differentiator called out in Phase 71 versus weaker image-to-3D baselines that skip a production-grade unwrap surface.

SF3D Illumination-Disentangled Texture Output#

Local3DSf3DIlluminationTexturePipeline promotes the released SF3D material channels into explicit provider assets so the lighting-neutral texture is available without scraping the final GLB:

ts
import {
  Local3DModelRegistry,
  createLocal3DSf3DIlluminationTexturePipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DSf3DIlluminationTexturePipeline({
  registry,
  runtimeRoot: '/opt/stable-fast-3d',
});

const result = await pipeline.run({
  images: [inputImageBuffer],
  textureResolution: 2048,
});

This task is grounded in the official Stability AI material assembly reviewed on 2026-03-26:

  • sf3d/system.py assigns decoded["features"] to mat_out["albedo"]
  • the same code path computes roughness, metallic, and tangent-space normal data before constructing trimesh.visual.material.PBRMaterial
  • the final GLB therefore already contains an illumination-neutral base color texture rather than a simple lit screenshot projection

The Oshun export surfaces those channels directly:

  • albedo is saved as a standalone texture asset and marked in metadata as albedoLightingDisentangled=true
  • normal is exported when present from the released material stack
  • scalar roughnessFactor and metallicFactor are preserved in metadata for downstream material authoring or validation
  • the textured glb remains part of the completion payload so the mesh and texture package stay consistent with the official SF3D runtime

This gives downstream tooling a proper lighting-neutral texture handoff, which is the Phase 71 differentiator here: not just “a textured mesh”, but the actual released albedo channel that SF3D decodes before material assembly.

Cost Dashboard#

Local3DCostDashboard can ingest tracked jobs from Local3DProviderRestApi and expose an API-first unit-economics snapshot:

ts
const costDashboard = createLocal3DCostDashboard();

const providerApi = createLocal3DProviderRestApi({
  capabilities: createLocal3DProviderCapabilities([
    'text-to-3d',
    'image-to-3d',
  ]),
  costDashboard,
});

Once jobs are flowing, query the dashboard at:

text
GET /v1/3d/costs/dashboard?selfHostedProfileId=runpod-l4-flex&cloudProfileIds=meshy-pro,meshy-studio

The default dashboard ships with:

  • self-hosted deployment profiles based on current RunPod serverless GPU pricing such as runpod-l4-flex and runpod-a40-flex
  • cloud API comparison profiles for Meshy Pro and Meshy Studio using current stage-credit pricing
  • per-task and aggregate comparisons showing self-hosted total cost, average cost per asset, estimated external API spend, and savings deltas

You can override both the self-hosted profiles and cloud pricing catalog when constructing the dashboard, so teams using on-prem GPUs, different storage retention windows, or other commercial APIs can plug in their own economics model without changing the REST contract.

Hunyuan Quality Dashboard#

Local3DHunyuan3D21QualityComparisonDashboard provides a deterministic, API-first benchmark snapshot for comparing the local Hunyuan3D 2.1 shape-plus-PBR stack against the current Meshy-6 PBR baseline surface:

ts
const hunyuanQualityDashboard =
  createLocal3DHunyuan3D21QualityComparisonDashboard();

const providerApi = createLocal3DProviderRestApi({
  capabilities: createLocal3DProviderCapabilities([
    'text-to-3d',
    'image-to-3d',
  ]),
  hunyuanQualityDashboard,
});

Query it at:

text
GET /v1/3d/quality/hunyuan-vs-meshy/dashboard?category=hard-surface&modality=image-to-3d-pbr&limit=3

The default snapshot ships with:

  • six curated text-to-3d + PBR and image-to-3d + PBR benchmark cases
  • weighted scoring across geometry fidelity, topology cleanliness, texture fidelity, material realism, prompt alignment, PBR completeness, lighting neutrality, and asset readiness
  • threshold gates that summarize whether the local Hunyuan stack still clears the target lead over Meshy-6 in average score, win rate, and PBR/material deltas
  • provider source references captured on 2026-03-26 from the official Tencent Hunyuan3D repository and current Meshy retexture API documentation

The dashboard is intentionally deterministic rather than a live online eval. It is designed for release gating, regression tracking, and API-level reporting, while still allowing teams to swap in their own benchmark dataset and scoring thresholds.

RunPod Serverless Proxy Mode#

For bursty workloads that should scale to zero between requests, the same local HTTP surface can now proxy jobs to a RunPod Serverless endpoint while keeping the Isis-compatible /v1/3d REST API and SSE progress stream online.

Set the following container environment variables:

text
LOCAL_3D_RUNPOD_ENDPOINT_ID=your-endpoint-id
LOCAL_3D_RUNPOD_API_KEY=your-runpod-api-key
LOCAL_3D_RUNPOD_SYNC_MODE=interactive
LOCAL_3D_RUNPOD_SYNC_TIMEOUT_SECONDS=90
LOCAL_3D_RUNPOD_EXECUTION_TIMEOUT=600
LOCAL_3D_RUNPOD_TTL=3600

When RunPod proxy mode is enabled:

  • interactive jobs can use /runsync while batch/background jobs use /run plus /status polling
  • local task IDs remain stable even though the adapter tracks a separate remote RunPod job ID internally
  • completed RunPod outputs can return encoded model/texture assets that the adapter hydrates back into local downloadable result assets
  • health checks treat zero-worker idle states as healthy, which matches RunPod's scale-to-zero behavior instead of reporting false failures

The adapter also exports request/result serializers (createLocal3DRunpodJobInput, parseLocal3DRunpodJobInput, and completion encode/decode helpers) so a RunPod worker can speak the same envelope format as the local gateway.

TripoSG Loader#

The package now also exposes Local3DTripoSGModelLoader for Tripo’s published 1.5B rectified-flow image-to-3D runtime:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGModelLoader,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const triposg = createLocal3DTripoSGModelLoader({ registry });
const plan = await triposg.resolveLoadPlan({
  gpuDevice: {
    id: 0,
    computeCapability: '8.9',
    vramBudgetMb: 12_288,
  },
});

This loader is grounded in Tripo’s published repo and Hugging Face artifacts:

  • VAST-AI/TripoSG publishes the diffusers TripoSGPipeline bundle with the Dinov2 encoder, rectified-flow transformer, scheduler, and VAE checkpoints
  • the official scripts.inference_triposg entrypoint also downloads briaai/RMBG-1.4 and runs BriaRMBG.from_pretrained(...) before prepare_image(...)
  • the upstream README publishes an 8GB CUDA minimum; Oshun keeps that as the hard floor and models ~12GB as an operational headroom target for the full TripoSG+RMBG runtime

The resolved load plan includes:

  • local paths for model_index.json, transformer, VAE, Dinov2 encoder, scheduler, and RMBG checkpoints
  • runtime defaults for num_inference_steps=50, guidance_scale=7.0, num_tokens=2048, and the published octree / flash-decoder settings
  • mixed-source dependency metadata covering the MIT TripoSG weights plus the separate BRIA RMBG 1.4 background-removal license
  • an explicit note that the underlying pipeline accepts list-image batches even though the released CLI only exposes a single-image entrypoint today

TripoSG Image-to-3D Pipeline#

The package now also exposes a runnable local TripoSG image-to-3D pipeline on top of the loader:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const triposgPipeline = createLocal3DTripoSGImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TripoSG',
});

const result = await triposgPipeline.run({
  images: [inputImageBuffer],
  quality: 'standard',
  outputFormat: 'glb',
  guidanceScale: 7,
  seed: 42,
});

This pipeline follows the released TripoSG flow closely:

  • it loads BriaRMBG from the separately downloaded briaai/RMBG-1.4 dependency and feeds the image through the published prepare_image(...) foreground-isolation path
  • it runs the official TripoSGPipeline.from_pretrained(...) stack in fp16 on CUDA with the published 50-step / 7.0-guidance defaults
  • it can optionally simplify the generated triangle mesh natively, and now automatically hands quad-dominant or targetPolyCount requests to MeshAnything V2 so downstream topology control is implemented instead of being warning-only intent

The resolved result includes:

  • geometry-only mesh assets in glb, obj, or ply format
  • the prepared foreground image used for inference so callers can inspect the actual crop fed into TripoSG
  • per-run metadata for preprocessing, inference, and optional simplification timings plus mesh vertex / face counts
  • explicit warnings when callers supply prompt, negative-prompt, or texture controls the released runtime cannot honor directly, plus MeshAnything post-process metadata when topology cleanup was applied automatically

TripoSG-Scribble Image-to-3D Pipeline#

The package also exposes a TripoSG-scribble pipeline for rough sketches and rapid concept iteration:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGScribbleImageTo3DPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const scribblePipeline = createLocal3DTripoSGScribbleImageTo3DPipeline({
  registry,
  runtimeRoot: '/opt/TripoSG',
});

const result = await scribblePipeline.run({
  images: [scribbleBuffer],
  prompt: 'a cat with wings',
  quality: 'standard',
  scribbleConfidence: 0.4,
  promptConfidence: 1.0,
});

This variant follows the published TripoSG-scribble release closely:

  • it downloads VAST-AI/TripoSG-scribble, which adds a tokenizer and CLIP text encoder on top of the TripoSG base image-conditioning stack
  • it keeps guidance_scale=0 because the upstream model is explicitly CFG-distilled and instead exposes prompt/scribble weighting through cross-attention scaling
  • it preserves the official use_flash_decoder=False and 8/8 octree settings from the released inference script so the local runtime does not reintroduce the boundary artifacts noted upstream
  • it automatically routes quad-dominant or targetPolyCount requests through MeshAnything V2 after the released scribble-conditioned mesh path finishes

The pipeline result includes:

  • geometry-only glb, obj, or ply outputs for rapid concept review
  • the prepared RGB sketch image that was actually fed into the model
  • per-run metadata for prompt text, confidence settings, denoising steps, timings, and mesh face / vertex counts
  • validation and warnings for unsupported negative-prompt and texturing controls, MeshAnything post-process metadata for topology cleanup requests, and guidance when scribble confidence moves outside the upstream-recommended 0.3-0.5 range

TripoSG Multi-View Consistency Mode#

The package also exposes a TripoSG multi-view consistency pipeline for 2-8 reference views of the same object:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGMultiViewConsistencyPipeline,
} from '@isis/3d-inference-local';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const pipeline = createLocal3DTripoSGMultiViewConsistencyPipeline({
  registry,
  runtimeRoot: '/opt/TripoSG',
});

const result = await pipeline.run({
  images: [frontBuffer, leftBuffer, backBuffer, rightBuffer],
  quality: 'standard',
  consistencySampleCount: 2048,
});

This mode is intentionally honest about what upstream TripoSG does and does not ship:

  • the released TripoSGPipeline accepts list-image batches, but it still reconstructs one mesh per image instead of doing true fused multi-view conditioning
  • Oshun therefore uses shared-seed per-view generation and then computes a pairwise surface-distance agreement score across the candidate meshes
  • the final output is the geometric medoid, which is the candidate mesh with the lowest mean pairwise Chamfer-style distance to the other reconstructed views
  • once that medoid is selected, Oshun can now automatically apply MeshAnything V2 when the request asks for quad-dominant topology or a face-budget target

The result includes:

  • the selected final glb, obj, or ply mesh
  • all prepared reference-view images used for conditioning
  • per-view consistency metrics, selected-view index, and cross-view agreement scores in metadata
  • automatic micro-batching when more than four views are supplied so Oshun stays inside the published TripoSG family’s practical VRAM envelope

TripoSG vs TripoSR Comparison#

The package also exposes a comparison harness for validating the local TripoSG surface against the repo’s older TripoSR-style provider baseline:

ts
import {
  Local3DModelRegistry,
  createLocal3DTripoSGComparisonPipeline,
} from '@isis/3d-inference-local';
import { createTripoProvider } from '@isis/3d-generation/providers';

const registry = new Local3DModelRegistry({
  rootDir: '/models/isis-3d',
});

const report = await createLocal3DTripoSGComparisonPipeline({
  registry,
  runtimeRoot: '/opt/TripoSG',
  baselineProvider: createTripoProvider({ apiKey: process.env.TRIPO_API_KEY }),
}).run({
  images: [referenceImageBuffer],
});

The report is intentionally explicit about evidence quality:

  • it runs the local TripoSG image-to-3D path and the injected Tripo baseline in parallel on the same request
  • it marks whether topology improvement was validated, mixed, baseline- competitive, or inconclusive
  • it assigns a confidence level based on whether baseline topology metrics came from provider metadata or a custom measured extractor
  • it separates texture parity from the topology verdict so the current geometry-only TripoSG surface is not misrepresented as a texture-complete replacement