Eve library

@isis/video-to-mesh

Other

Revision 1

On this page

Video-first 3D reconstruction and multi-view meshing workflows for Isis.

Status#

This package now ships the first 71.4 reconstruction and scanning stages:

  • monocular-video-depth-estimation
  • video-camera-pose-estimation
  • multi-frame-depth-fusion
  • point-cloud-surface-reconstruction
  • video-texture-projection
  • video-object-segmentation
  • turntable-video-reconstruction
  • video-quality-assessment
  • progressive-reconstruction-preview
  • colmap-structure-from-motion
  • dense-multi-view-stereo
  • neural-multi-view-reconstruction
  • guided-capture-assistant
  • smartphone-capture-streaming
  • 4dgs-temporal-mesh-export
  • temporally-coherent-gaussian-mesh-extraction
  • 4dgs-blend-shape-conversion
  • 4dgs-streaming-compression
  • 4dgs-editing
  • visual-slam
  • lidar-fusion
  • slam-to-gaussian-splatting
  • room-scale-scanning
  • object-scale-scanning

The depth-estimation workflow produces per-frame relative depth maps from a source video using Depth Anything V2, with:

  • official Depth Anything V2 runtime support
  • Hugging Face Transformers fallback
  • raw .npy depth tensors per sampled frame
  • 16-bit PNG and colorized PNG exports
  • optional preview video generation
  • explicit commercial-safe defaulting to the Small Apache-2.0 checkpoint

The camera-pose workflow runs sequential COLMAP SfM and exports the selected text model needed for downstream sparse-point calibration.

The new colmap-structure-from-motion workflow extends COLMAP integration to true multi-view still-image captures. It accepts prepared image manifests, images directories, or explicit image-path lists; resolves an honest matcher strategy (exhaustive for unordered sets, sequential for ordered orbit or sequence captures); and exports the selected sparse model, text-model conversion, per-image intrinsics and extrinsics, plus registration status for every staged image.

The new dense-multi-view-stereo workflow consumes a sparse COLMAP model and runs the documented dense path of image_undistorter, patch_match_stereo, and stereo_fusion. It exports the dense COLMAP workspace, per-view depth and normal maps, consistency graphs, and a fused point cloud instead of pretending that sparse SfM is already dense geometry.

The new neural-multi-view-reconstruction workflow adds a real local fast neural object-reconstruction route on top of the shipped @isis/3d-inference-local TRELLIS.2 multi-image backend. It accepts generic image manifests, segmentation manifests, camera-pose outputs, COLMAP SfM manifests, image directories, or explicit image-path lists; deterministically downsamples larger captures to the backend's supported 2-6 views; and returns an .obj or .ply mesh plus a manifest that records which views were selected and how the neural backend conditioned on them. This is explicitly an InstantMesh-equivalent local integration rather than a fake placeholder: the current repo does not ship an InstantMesh runtime, so the workflow uses the existing multi-image TRELLIS.2 checkpoint that is already integrated elsewhere in Isis.

The new guided-capture-assistant workflow consumes quality-assessment, camera-pose, COLMAP SfM, segmentation, or plain frame manifests and turns them into concrete next-shot instructions. When camera poses are available it infers orbit coverage gaps and tells the operator exactly how far to rotate left or right, whether a top or low-angle shot is still missing, and where overlap needs to be increased; when only quality or segmentation artifacts exist it still issues honest fallback guidance about blur, lighting, framing, and orbit completeness.

The new smartphone-capture-streaming workflow turns a phone browser into a live multi-view capture client. It emits a browser bootstrap that prefers the rear camera through getUserMedia / ImageCapture, accepts binary or base64 frame uploads, keeps both frame and image manifests up to date, periodically runs the shipped quality-assessment and guided-capture stages on the growing session, and triggers fast neural preview reconstructions so operators can see rolling mesh checkpoints while still capturing.

The new 4dgs-temporal-mesh-export workflow converts a temporal Gaussian field into a reusable animated mesh package using the same anchor-driven proxy-mesh strategy already explored elsewhere in Isis 4DGS research. It can read a serialized FourDGSTrainingArtifact JSON or an in-memory artifact object, writes a bind-pose .obj, exports a sampled .obj mesh sequence with consistent topology across frames, and emits a rig/keyframe JSON manifest for downstream glTF/FBX or skeletal-animation toolchains.

The new temporally-coherent-gaussian-mesh-extraction workflow focuses specifically on topology-locked surface extraction from a trained temporal Gaussian field. It starts from the same canonical proxy mesh, samples the deforming surface across time, then applies temporal smoothing and edge-stretch regularization so every extracted frame keeps the same vertex and triangle layout while staying within explicit coherence bounds. It writes a canonical reference mesh, a coherent per-frame .obj sequence, a vertex-trajectory manifest, and a coherence report with stretch, velocity, acceleration, and flagged-frame metrics.

The new 4dgs-blend-shape-conversion workflow converts a trained temporal Gaussian field into a reusable morph-target package. It samples the same topology-preserving proxy mesh over time, applies temporal smoothing, resolves whether the dominant motion is facial or body-driven, and then decomposes the observed deformations into sparse blend-shape targets plus normalized per-frame weight tracks. The output includes a neutral base mesh, optional target .obj meshes, a weight-track JSON sidecar, and a manifest that records explained variance, reconstruction error, and region labels for each extracted target.

The new 4dgs-streaming-compression workflow packages a trained temporal Gaussian field for actual streaming playback rather than offline inspection only. It analyzes anchor motion over time, prunes low-value motion under an explicit delivery profile, emits quantized keyframes plus temporal delta blocks, partitions anchors into spatial stream chunks, and writes a compression report, streaming manifest, per-chunk payloads, bootstrap plan, and decode-preview estimate. The package is honest about tradeoffs: mobile, web, desktop, and archive profiles expose different bitrate, startup-latency, and quality retention targets instead of pretending one codec setting fits every client.

The new 4dgs-editing workflow turns the same trained temporal Gaussian field into an authoring-friendly editing package. It derives semantic labels from anchor identities, scores anchor stability and motion, resolves weighted selections from anchor ids, semantic labels, or 3D bounds, and then applies non-destructive transform, appearance, motion, duplication, and hide operations. It writes a workspace manifest, edited-scene manifest, and preview-frame sample sequence so animated edits can be validated before any downstream export or streaming pass.

The new depth-fusion workflow consumes the depth manifest plus the COLMAP camera-pose manifest, fits each relative depth map onto the COLMAP sparse scene scale, then back-projects and voxel-merges a colored point cloud in COLMAP world coordinates. The result is scene-scale consistent but still not metric, because monocular depth and COLMAP reconstruction remain scale-ambiguous.

The new surface-reconstruction workflow consumes the fused point cloud and runs Open3D Poisson surface reconstruction, including automatic normal estimation/orientation when the cloud lacks usable normals, density-based vertex trimming, and small disconnected-component cleanup. It exports untextured .ply and .obj meshes plus optional raw Poisson density values.

The new video-texture-projection workflow consumes the reconstructed mesh plus the camera-pose manifest, ray-tests mesh visibility from each source frame, and projects view-weighted vertex colors back onto the mesh. It exports a vertex-colored .ply/.obj mesh and fills unobserved vertices from neighboring colored regions instead of pretending to ship a UV-baked texture atlas.

The new video-object-segmentation workflow stages a video or frame sequence, runs the official SAM 2 video predictor with auto-central, box, or point prompts, propagates a single-object foreground mask across the clip, and writes binary masks plus masked RGB cutout frames for every sampled frame. Its manifest is directly reusable as a camera-pose frame manifest because each frame entry's sourceFramePath points at the masked RGB cutout, and it can also emit a masked preview .mp4 that the current monocular-depth workflow can ingest as a segmented video source.

The new turntable-video-reconstruction workflow specializes for the stationary camera plus rotating-object capture pattern. It consumes a segmentation manifest or runs inline SAM 2 segmentation from a raw input video, selects evenly spaced object views over the rotation, synthesizes an orbit-camera rig, carves a silhouette visual hull, extracts a Poisson surface, and projects masked frame colors back onto the mesh as vertex colors. It is honest about the method: this path reconstructs a silhouette-consistent object mesh under a turntable assumption instead of pretending to recover generic moving-camera SfM geometry.

The new video-quality-assessment workflow samples a raw video, generic frame manifest, or segmentation manifest and scores whether the capture is suitable for reconstruction. It measures viewpoint novelty for coverage, Laplacian and gradient energy for blur, luminance/dynamic-range/clipping for lighting, emits per-frame diagnostics plus recommendations, and writes a sampled-frame manifest that downstream reconstruction stages can reuse directly.

The new progressive-reconstruction-preview workflow reuses the real local depth, pose, fusion, and Poisson surface stages at growing frame-count checkpoints, renders point-cloud or mesh snapshots for each checkpoint, and assembles a preview video that shows the reconstruction stabilizing as more frames are incorporated. It can start from a raw input video or from prepared depth and camera-pose manifests.

The broader reconstruction roadmap remains encoded in the package manifest and mirrors the 71.4 task breakdown in TODOS.md.

The new lidar-fusion workflow ingests Apple ARKit sceneDepth or smoothedSceneDepth LiDAR frames from supported iPhone Pro and iPad Pro captures, applies confidence-aware hole filling plus temporal stabilization, and fuses the resulting metric depth into an ARKit-world point cloud. It writes per-frame processed depth artifacts, a camera-trajectory sidecar, a confidence coverage report, and a fused .ply point cloud that later SLAM-to-Gaussian, room-scale, and object-scale scanning stages can reuse directly.

The new slam-to-gaussian-splatting workflow converts the shipped visual-slam artifacts into a reusable Gaussian scene. When a shipped lidar-fusion manifest is also present, it estimates a similarity transform between the SLAM trajectory and the ARKit LiDAR trajectory, lifts the sparse landmark map into metric ARKit world coordinates, and combines aligned landmark anchors with the fused LiDAR point cloud. It writes an alignment report, aligned trajectory and landmark sidecars, a compact scene-artifact JSON, a Graphdeco-style Gaussian .ply, and an Isis .isplat binary for downstream room-scale and object-scale scanning stages.

The new room-scale-scanning workflow composes the shipped visual-slam, lidar-fusion, slam-to-gaussian-splatting, and Poisson point-cloud-surface-reconstruction stages into a single room capture flow. It can start from raw stereo-observation plus LiDAR sequences or from persisted SLAM and LiDAR manifests, exports both a full-room mesh and a Gaussian scene, and writes a room report that estimates the room footprint, height, volume, floor/wall/ceiling coverage, walkthrough traversal quality, and concrete next capture actions when the scan is still partial.

The new object-scale-scanning workflow applies the shipped SLAM + LiDAR stack to a single foreground object instead of a room envelope. It estimates the object center from the capture orbit, crops the fused LiDAR cloud to the dominant foreground cluster, trims support-surface spill when a tabletop or pedestal is detected, reconstructs a cleaned mesh from that filtered point cloud, and emits concrete follow-up guidance for missing orbit, top-view, or low-angle coverage.

Current Workflow#

Use estimateLocal3DVideoToMeshMonocularVideoDepth(...) or Local3DVideoToMeshMonocularDepthPipeline for the depth-estimation stage.

Use estimateLocal3DVideoToMeshCameraPoses(...) or Local3DVideoToMeshCameraPosePipeline for the COLMAP sequential SfM stage. The camera-pose workflow can ingest a raw input video, a frame manifest emitted by the depth-estimation stage, or an existing frames directory.

Use estimateLocal3DVideoToMeshColmapStructureFromMotion(...) or Local3DVideoToMeshColmapSfMPipeline for still-image multi-view COLMAP SfM. The workflow can ingest a prepared image manifest, a directory of stills, or an explicit list of image paths; it exports the chosen sparse model, the converted text model, registered and unregistered image summaries, and per-image camera poses for the best sparse component.

Use reconstructLocal3DVideoToMeshDenseMvs(...) or Local3DVideoToMeshDenseMvsPipeline to densify a sparse COLMAP model into a fused dense point cloud. The workflow accepts the shipped sparse manifest directly, but it can also start from an explicit imagesDir plus sparseModelDir. It exports the dense COLMAP workspace, per-view depth and normal maps, consistency graphs, and a fused .ply point cloud.

Use reconstructLocal3DVideoToMeshNeuralMultiView(...) or Local3DVideoToMeshNeuralMultiViewReconstructionPipeline when you want a fast feed-forward object mesh from a sparse image set instead of classical SfM plus dense stereo. The workflow accepts a generic manifest, images directory, or explicit image paths; automatically reuses SAM2 segmentation manifests and COLMAP manifests; and packages the selected neural input views together with the final .obj or .ply mesh.

Use guideLocal3DVideoToMeshCapture(...) or Local3DVideoToMeshGuidedCaptureAssistantPipeline when you need concrete capture instructions before continuing a multi-view session. The workflow can combine quality, SfM, segmentation, and frame manifests; emits a persisted capture-guidance report; and prioritizes actions such as rotating 15 degrees to close an orbit gap, tilting down to recenter the object, adding a top view, or stabilizing the camera before more images are taken.

Use createLocal3DVideoToMeshSmartphoneCaptureSession(...) or Local3DVideoToMeshSmartphoneCaptureStreamingPipeline when capture originates from a phone. The session exposes a browser bootstrap payload, writes every accepted frame into reusable manifests, serves optional Node HTTP upload/status handlers, checkpoints quality and guidance artifacts as more frames arrive, and can call the fast neural multi-view reconstruction route for rolling preview meshes during the session.

Use exportLocal3DVideoToMesh4DGSTemporalMesh(...) or Local3DVideoToMesh4DGSTemporalMeshExportPipeline when a trained FourDGSTrainingArtifact already exists and you need an animated mesh export. The workflow accepts either an in-memory artifact object or a serialized JSON artifact path, writes a bind-pose .obj, optionally writes sampled per-frame .obj meshes, and persists a rig manifest containing joints, weights, triangles, and keyframes.

Use extractLocal3DVideoToMeshTemporallyCoherentGaussianMesh(...) or Local3DVideoToMeshTemporallyCoherentGaussianMeshExtractionPipeline when the priority is a temporally coherent mesh sequence rather than a skeletal package. This workflow accepts the same trained FourDGSTrainingArtifact inputs, emits a canonical reference .obj, writes a topology-preserving coherent .obj sequence, and optionally persists vertex-trajectory and coherence-report JSON sidecars for downstream mesh QA or motion-analysis steps.

Use convertLocal3DVideoToMesh4DGSBlendShapes(...) or Local3DVideoToMesh4DGSBlendShapeConversionPipeline when the output needs to be blend shapes rather than a mesh sequence. The workflow accepts the same trained FourDGSTrainingArtifact inputs, emits a neutral base .obj, optionally writes individual target .obj meshes, and persists normalized per-frame weight tracks that downstream avatar or facial-retargeting systems can ingest.

Use compressLocal3DVideoToMesh4DGSForStreaming(...) or Local3DVideoToMesh4DGSStreamingCompressionPipeline when the trained FourDGSTrainingArtifact needs to be shipped as a bandwidth-aware animated Gaussian package. The workflow accepts in-memory or JSON artifacts, writes a compression report, streaming manifest, chunk payload directory, optional bootstrap plan, and optional decode-preview estimate, and supports separate mobile, web, desktop, and archive delivery profiles.

Use editLocal3DVideoToMesh4DGS(...) or Local3DVideoToMesh4DGSEditingPipeline when you need non-destructive animated Gaussian scene modification before streaming or export. The workflow accepts in-memory or JSON artifacts plus a sequence of edit operations, emits a semantic workspace, writes the edited anchor-state manifest, and optionally samples preview frames across time so transform, appearance, motion, duplicate, and hide operations can be inspected locally.

Use fuseLocal3DVideoToMeshMultiFrameDepth(...) or Local3DVideoToMeshDepthFusionPipeline to align the per-frame relative depth maps against COLMAP sparse points and emit a fused .ply point cloud plus an optional .npz point cache.

Use reconstructLocal3DVideoToMeshSurface(...) or Local3DVideoToMeshSurfaceReconstructionPipeline to convert the fused point cloud into an untextured mesh with Poisson surface reconstruction.

Use projectLocal3DVideoToMeshTexture(...) or Local3DVideoToMeshTextureProjectionPipeline to project source-frame colors onto the reconstructed mesh as vertex colors with raycast visibility checks.

Use segmentLocal3DVideoToMeshObject(...) or Local3DVideoToMeshObjectSegmentationPipeline to isolate a foreground object before reconstruction. The workflow accepts a raw video, a frame manifest, or a frames directory; defaults to SAM 2 small with auto-central prompting; and produces mask PNGs, masked RGB frames, optional RGBA cutouts, and an optional masked preview video.

Use reconstructLocal3DVideoToMeshTurntable(...) or Local3DVideoToMeshTurntableReconstructionPipeline when the object is rotating on a turntable while the camera stays mostly fixed. The workflow can start from a raw video and run inline segmentation, or consume the segmentation manifest from the prior workflow directly. It exports selected-view and synthetic-pose manifests, a carved point cloud, an untextured mesh, and a vertex-colored mesh.

Use assessLocal3DVideoToMeshVideoQuality(...) or Local3DVideoToMeshVideoQualityAssessmentPipeline before reconstruction when you need to detect insufficient viewpoint coverage, motion blur, or poor lighting. The workflow accepts a raw video, a frame manifest, or a segmentation manifest; exports a reusable sampled-frame manifest; and returns coverage, sharpness, lighting, issue, and recommendation summaries.

Use previewLocal3DVideoToMeshProgressiveReconstruction(...) or Local3DVideoToMeshProgressiveReconstructionPreviewPipeline when you want checkpointed previews of the reconstruction forming over time. The workflow can run inline from a raw video or consume prepared depth and pose manifests, emits checkpoint point clouds or meshes plus preview PNGs, and assembles an .mp4 timeline that clients can display as a progressive reconstruction preview.

Use trackLocal3DVideoToMeshVisualSlam(...), Local3DVideoToMeshVisualSlamPipeline, or Local3DVideoToMeshVisualSlamSession when you need real-time stereo camera tracking plus sparse landmark mapping. The shipped workflow is observation driven: it accepts an in-memory stereo feature sequence or a JSON manifest of stereo observations, triangulates sparse landmarks, estimates per-frame poses, detects place-signature loop closures, supports localization-only replay from a serialized map, and writes reusable trajectory, landmark, loop-closure, map-snapshot, and localization-metrics artifacts for downstream SLAM and Gaussian-conversion tasks.

Use fuseLocal3DVideoToMeshLidar(...) or Local3DVideoToMeshLidarFusionPipeline when capture originates from ARKit LiDAR hardware. The workflow accepts either an in-memory ARKit LiDAR frame sequence or a JSON manifest of frames containing metric sceneDepth / smoothedSceneDepth, confidence maps, intrinsics, and camera transforms. It outputs processed per-frame metric depth artifacts, a trajectory manifest, a confidence report, and a fused ARKit-world .ply point cloud instead of pretending monocular relative depth is already metric.

Use convertLocal3DVideoToMeshSlamToGaussianSplatting(...) or Local3DVideoToMeshSlamToGaussianSplattingPipeline when you need a Gaussian scene directly from the shipped SLAM stack. The workflow accepts a shipped visual-slam manifest and can optionally accept a shipped lidar-fusion manifest; if LiDAR is present, it aligns the SLAM trajectory into ARKit world coordinates with a similarity fit, exports aligned sparse-map sidecars, and packages the final Gaussian scene as both Graphdeco-style .ply and Isis .isplat artifacts. Without LiDAR it still exports a sparse Gaussian anchor scene in visual-SLAM coordinates, but the output remains non-metric and the colorization is confidence-coded rather than photometric.

Use scanLocal3DVideoToMeshRoomScale(...) or Local3DVideoToMeshRoomScaleScanningPipeline when you want a full room scan from the shipped SLAM stack instead of only intermediate artifacts. The workflow can run inline from stereo observations plus ARKit LiDAR frames or reuse shipped visual-slam and lidar-fusion manifests directly; it exports a Poisson room mesh, a metric Gaussian scene, and a room report that estimates room bounds, footprint area, room height and volume, floor/wall/ceiling coverage, traversal completeness, and follow-up capture guidance.

Use scanLocal3DVideoToMeshObjectScale(...) or Local3DVideoToMeshObjectScaleScanningPipeline when the target is a single object captured with an ARKit LiDAR device. The workflow accepts raw stereo + LiDAR sequences or the shipped visual-slam and lidar-fusion manifests, optionally reuses segmentation framing signals, writes a cleaned foreground .ply, reconstructs an object mesh from that filtered cloud, and persists a guidance report describing the exact views still worth capturing before final export.

Expected Python-side runtime dependencies:

  • torch
  • opencv-python
  • numpy
  • official Depth-Anything-V2 checkout or installed depth_anything_v2 module for the preferred backend
  • transformers for fallback loading via Hugging Face
  • colmap for camera-pose estimation
  • opencv-python for frame and depth image loading during fusion
  • numpy for sparse-calibrated point-cloud fusion
  • open3d for Poisson surface reconstruction and normal estimation
  • opencv-python for source-frame sampling during texture projection
  • official sam2 package for promptable video object segmentation
  • open3d for visual-hull carving, Poisson extraction, and mesh coloring in turntable reconstruction
  • opencv-python and numpy for capture-quality metrics and sampled-frame staging in video quality assessment
  • open3d, opencv-python, and numpy for checkpoint preview rendering and progressive preview video assembly
  • @isis/3d-inference-local with its TRELLIS.2 multi-image runtime dependencies for the neural multi-view reconstruction workflow

Planned Tracks#

  • monocular-video-to-3d
  • multi-view-reconstruction
  • 4d-gaussian-splatting-to-animated-mesh

Supported Tracks#

  • slam-based-3d-scanning

Manifest#

Use createLocal3DVideoToMeshPackageManifest() to inspect the current package status plus the remaining planned 71.4 roadmap surface without overstating support.