# @isis/video-to-mesh Video-first 3D reconstruction and multi-view meshing workflows for Isis. ## Status This package now ships the first `71.4` reconstruction and scanning stages: - `monocular-video-depth-estimation` - `video-camera-pose-estimation` - `multi-frame-depth-fusion` - `point-cloud-surface-reconstruction` - `video-texture-projection` - `video-object-segmentation` - `turntable-video-reconstruction` - `video-quality-assessment` - `progressive-reconstruction-preview` - `colmap-structure-from-motion` - `dense-multi-view-stereo` - `neural-multi-view-reconstruction` - `guided-capture-assistant` - `smartphone-capture-streaming` - `4dgs-temporal-mesh-export` - `temporally-coherent-gaussian-mesh-extraction` - `4dgs-blend-shape-conversion` - `4dgs-streaming-compression` - `4dgs-editing` - `visual-slam` - `lidar-fusion` - `slam-to-gaussian-splatting` - `room-scale-scanning` - `object-scale-scanning` The depth-estimation workflow produces per-frame relative depth maps from a source video using Depth Anything V2, with: - official Depth Anything V2 runtime support - Hugging Face Transformers fallback - raw `.npy` depth tensors per sampled frame - 16-bit PNG and colorized PNG exports - optional preview video generation - explicit commercial-safe defaulting to the Small Apache-2.0 checkpoint The camera-pose workflow runs sequential COLMAP SfM and exports the selected text model needed for downstream sparse-point calibration. The new `colmap-structure-from-motion` workflow extends COLMAP integration to true multi-view still-image captures. It accepts prepared image manifests, images directories, or explicit image-path lists; resolves an honest matcher strategy (`exhaustive` for unordered sets, `sequential` for ordered orbit or sequence captures); and exports the selected sparse model, text-model conversion, per-image intrinsics and extrinsics, plus registration status for every staged image. The new `dense-multi-view-stereo` workflow consumes a sparse COLMAP model and runs the documented dense path of `image_undistorter`, `patch_match_stereo`, and `stereo_fusion`. It exports the dense COLMAP workspace, per-view depth and normal maps, consistency graphs, and a fused point cloud instead of pretending that sparse SfM is already dense geometry. The new `neural-multi-view-reconstruction` workflow adds a real local fast neural object-reconstruction route on top of the shipped `@isis/3d-inference-local` TRELLIS.2 multi-image backend. It accepts generic image manifests, segmentation manifests, camera-pose outputs, COLMAP SfM manifests, image directories, or explicit image-path lists; deterministically downsamples larger captures to the backend's supported 2-6 views; and returns an `.obj` or `.ply` mesh plus a manifest that records which views were selected and how the neural backend conditioned on them. This is explicitly an InstantMesh-equivalent local integration rather than a fake placeholder: the current repo does not ship an InstantMesh runtime, so the workflow uses the existing multi-image TRELLIS.2 checkpoint that is already integrated elsewhere in Isis. The new `guided-capture-assistant` workflow consumes quality-assessment, camera-pose, COLMAP SfM, segmentation, or plain frame manifests and turns them into concrete next-shot instructions. When camera poses are available it infers orbit coverage gaps and tells the operator exactly how far to rotate left or right, whether a top or low-angle shot is still missing, and where overlap needs to be increased; when only quality or segmentation artifacts exist it still issues honest fallback guidance about blur, lighting, framing, and orbit completeness. The new `smartphone-capture-streaming` workflow turns a phone browser into a live multi-view capture client. It emits a browser bootstrap that prefers the rear camera through `getUserMedia` / `ImageCapture`, accepts binary or base64 frame uploads, keeps both frame and image manifests up to date, periodically runs the shipped quality-assessment and guided-capture stages on the growing session, and triggers fast neural preview reconstructions so operators can see rolling mesh checkpoints while still capturing. The new `4dgs-temporal-mesh-export` workflow converts a temporal Gaussian field into a reusable animated mesh package using the same anchor-driven proxy-mesh strategy already explored elsewhere in Isis 4DGS research. It can read a serialized `FourDGSTrainingArtifact` JSON or an in-memory artifact object, writes a bind-pose `.obj`, exports a sampled `.obj` mesh sequence with consistent topology across frames, and emits a rig/keyframe JSON manifest for downstream glTF/FBX or skeletal-animation toolchains. The new `temporally-coherent-gaussian-mesh-extraction` workflow focuses specifically on topology-locked surface extraction from a trained temporal Gaussian field. It starts from the same canonical proxy mesh, samples the deforming surface across time, then applies temporal smoothing and edge-stretch regularization so every extracted frame keeps the same vertex and triangle layout while staying within explicit coherence bounds. It writes a canonical reference mesh, a coherent per-frame `.obj` sequence, a vertex-trajectory manifest, and a coherence report with stretch, velocity, acceleration, and flagged-frame metrics. The new `4dgs-blend-shape-conversion` workflow converts a trained temporal Gaussian field into a reusable morph-target package. It samples the same topology-preserving proxy mesh over time, applies temporal smoothing, resolves whether the dominant motion is facial or body-driven, and then decomposes the observed deformations into sparse blend-shape targets plus normalized per-frame weight tracks. The output includes a neutral base mesh, optional target `.obj` meshes, a weight-track JSON sidecar, and a manifest that records explained variance, reconstruction error, and region labels for each extracted target. The new `4dgs-streaming-compression` workflow packages a trained temporal Gaussian field for actual streaming playback rather than offline inspection only. It analyzes anchor motion over time, prunes low-value motion under an explicit delivery profile, emits quantized keyframes plus temporal delta blocks, partitions anchors into spatial stream chunks, and writes a compression report, streaming manifest, per-chunk payloads, bootstrap plan, and decode-preview estimate. The package is honest about tradeoffs: mobile, web, desktop, and archive profiles expose different bitrate, startup-latency, and quality retention targets instead of pretending one codec setting fits every client. The new `4dgs-editing` workflow turns the same trained temporal Gaussian field into an authoring-friendly editing package. It derives semantic labels from anchor identities, scores anchor stability and motion, resolves weighted selections from anchor ids, semantic labels, or 3D bounds, and then applies non-destructive transform, appearance, motion, duplication, and hide operations. It writes a workspace manifest, edited-scene manifest, and preview-frame sample sequence so animated edits can be validated before any downstream export or streaming pass. The new depth-fusion workflow consumes the depth manifest plus the COLMAP camera-pose manifest, fits each relative depth map onto the COLMAP sparse scene scale, then back-projects and voxel-merges a colored point cloud in COLMAP world coordinates. The result is scene-scale consistent but still not metric, because monocular depth and COLMAP reconstruction remain scale-ambiguous. The new surface-reconstruction workflow consumes the fused point cloud and runs Open3D Poisson surface reconstruction, including automatic normal estimation/orientation when the cloud lacks usable normals, density-based vertex trimming, and small disconnected-component cleanup. It exports untextured `.ply` and `.obj` meshes plus optional raw Poisson density values. The new video-texture-projection workflow consumes the reconstructed mesh plus the camera-pose manifest, ray-tests mesh visibility from each source frame, and projects view-weighted vertex colors back onto the mesh. It exports a vertex-colored `.ply`/`.obj` mesh and fills unobserved vertices from neighboring colored regions instead of pretending to ship a UV-baked texture atlas. The new video-object-segmentation workflow stages a video or frame sequence, runs the official SAM 2 video predictor with auto-central, box, or point prompts, propagates a single-object foreground mask across the clip, and writes binary masks plus masked RGB cutout frames for every sampled frame. Its manifest is directly reusable as a camera-pose frame manifest because each frame entry's `sourceFramePath` points at the masked RGB cutout, and it can also emit a masked preview `.mp4` that the current monocular-depth workflow can ingest as a segmented video source. The new turntable-video-reconstruction workflow specializes for the stationary camera plus rotating-object capture pattern. It consumes a segmentation manifest or runs inline SAM 2 segmentation from a raw input video, selects evenly spaced object views over the rotation, synthesizes an orbit-camera rig, carves a silhouette visual hull, extracts a Poisson surface, and projects masked frame colors back onto the mesh as vertex colors. It is honest about the method: this path reconstructs a silhouette-consistent object mesh under a turntable assumption instead of pretending to recover generic moving-camera SfM geometry. The new video-quality-assessment workflow samples a raw video, generic frame manifest, or segmentation manifest and scores whether the capture is suitable for reconstruction. It measures viewpoint novelty for coverage, Laplacian and gradient energy for blur, luminance/dynamic-range/clipping for lighting, emits per-frame diagnostics plus recommendations, and writes a sampled-frame manifest that downstream reconstruction stages can reuse directly. The new progressive-reconstruction-preview workflow reuses the real local depth, pose, fusion, and Poisson surface stages at growing frame-count checkpoints, renders point-cloud or mesh snapshots for each checkpoint, and assembles a preview video that shows the reconstruction stabilizing as more frames are incorporated. It can start from a raw input video or from prepared depth and camera-pose manifests. The broader reconstruction roadmap remains encoded in the package manifest and mirrors the `71.4` task breakdown in `TODOS.md`. The new `lidar-fusion` workflow ingests Apple ARKit `sceneDepth` or `smoothedSceneDepth` LiDAR frames from supported iPhone Pro and iPad Pro captures, applies confidence-aware hole filling plus temporal stabilization, and fuses the resulting metric depth into an ARKit-world point cloud. It writes per-frame processed depth artifacts, a camera-trajectory sidecar, a confidence coverage report, and a fused `.ply` point cloud that later SLAM-to-Gaussian, room-scale, and object-scale scanning stages can reuse directly. The new `slam-to-gaussian-splatting` workflow converts the shipped `visual-slam` artifacts into a reusable Gaussian scene. When a shipped `lidar-fusion` manifest is also present, it estimates a similarity transform between the SLAM trajectory and the ARKit LiDAR trajectory, lifts the sparse landmark map into metric ARKit world coordinates, and combines aligned landmark anchors with the fused LiDAR point cloud. It writes an alignment report, aligned trajectory and landmark sidecars, a compact scene-artifact JSON, a Graphdeco-style Gaussian `.ply`, and an Isis `.isplat` binary for downstream room-scale and object-scale scanning stages. The new `room-scale-scanning` workflow composes the shipped `visual-slam`, `lidar-fusion`, `slam-to-gaussian-splatting`, and Poisson `point-cloud-surface-reconstruction` stages into a single room capture flow. It can start from raw stereo-observation plus LiDAR sequences or from persisted SLAM and LiDAR manifests, exports both a full-room mesh and a Gaussian scene, and writes a room report that estimates the room footprint, height, volume, floor/wall/ceiling coverage, walkthrough traversal quality, and concrete next capture actions when the scan is still partial. The new `object-scale-scanning` workflow applies the shipped SLAM + LiDAR stack to a single foreground object instead of a room envelope. It estimates the object center from the capture orbit, crops the fused LiDAR cloud to the dominant foreground cluster, trims support-surface spill when a tabletop or pedestal is detected, reconstructs a cleaned mesh from that filtered point cloud, and emits concrete follow-up guidance for missing orbit, top-view, or low-angle coverage. ## Current Workflow Use `estimateLocal3DVideoToMeshMonocularVideoDepth(...)` or `Local3DVideoToMeshMonocularDepthPipeline` for the depth-estimation stage. Use `estimateLocal3DVideoToMeshCameraPoses(...)` or `Local3DVideoToMeshCameraPosePipeline` for the COLMAP sequential SfM stage. The camera-pose workflow can ingest a raw input video, a frame manifest emitted by the depth-estimation stage, or an existing frames directory. Use `estimateLocal3DVideoToMeshColmapStructureFromMotion(...)` or `Local3DVideoToMeshColmapSfMPipeline` for still-image multi-view COLMAP SfM. The workflow can ingest a prepared image manifest, a directory of stills, or an explicit list of image paths; it exports the chosen sparse model, the converted text model, registered and unregistered image summaries, and per-image camera poses for the best sparse component. Use `reconstructLocal3DVideoToMeshDenseMvs(...)` or `Local3DVideoToMeshDenseMvsPipeline` to densify a sparse COLMAP model into a fused dense point cloud. The workflow accepts the shipped sparse manifest directly, but it can also start from an explicit `imagesDir` plus `sparseModelDir`. It exports the dense COLMAP workspace, per-view depth and normal maps, consistency graphs, and a fused `.ply` point cloud. Use `reconstructLocal3DVideoToMeshNeuralMultiView(...)` or `Local3DVideoToMeshNeuralMultiViewReconstructionPipeline` when you want a fast feed-forward object mesh from a sparse image set instead of classical SfM plus dense stereo. The workflow accepts a generic manifest, images directory, or explicit image paths; automatically reuses SAM2 segmentation manifests and COLMAP manifests; and packages the selected neural input views together with the final `.obj` or `.ply` mesh. Use `guideLocal3DVideoToMeshCapture(...)` or `Local3DVideoToMeshGuidedCaptureAssistantPipeline` when you need concrete capture instructions before continuing a multi-view session. The workflow can combine quality, SfM, segmentation, and frame manifests; emits a persisted capture-guidance report; and prioritizes actions such as rotating 15 degrees to close an orbit gap, tilting down to recenter the object, adding a top view, or stabilizing the camera before more images are taken. Use `createLocal3DVideoToMeshSmartphoneCaptureSession(...)` or `Local3DVideoToMeshSmartphoneCaptureStreamingPipeline` when capture originates from a phone. The session exposes a browser bootstrap payload, writes every accepted frame into reusable manifests, serves optional Node HTTP upload/status handlers, checkpoints quality and guidance artifacts as more frames arrive, and can call the fast neural multi-view reconstruction route for rolling preview meshes during the session. Use `exportLocal3DVideoToMesh4DGSTemporalMesh(...)` or `Local3DVideoToMesh4DGSTemporalMeshExportPipeline` when a trained `FourDGSTrainingArtifact` already exists and you need an animated mesh export. The workflow accepts either an in-memory artifact object or a serialized JSON artifact path, writes a bind-pose `.obj`, optionally writes sampled per-frame `.obj` meshes, and persists a rig manifest containing joints, weights, triangles, and keyframes. Use `extractLocal3DVideoToMeshTemporallyCoherentGaussianMesh(...)` or `Local3DVideoToMeshTemporallyCoherentGaussianMeshExtractionPipeline` when the priority is a temporally coherent mesh sequence rather than a skeletal package. This workflow accepts the same trained `FourDGSTrainingArtifact` inputs, emits a canonical reference `.obj`, writes a topology-preserving coherent `.obj` sequence, and optionally persists vertex-trajectory and coherence-report JSON sidecars for downstream mesh QA or motion-analysis steps. Use `convertLocal3DVideoToMesh4DGSBlendShapes(...)` or `Local3DVideoToMesh4DGSBlendShapeConversionPipeline` when the output needs to be blend shapes rather than a mesh sequence. The workflow accepts the same trained `FourDGSTrainingArtifact` inputs, emits a neutral base `.obj`, optionally writes individual target `.obj` meshes, and persists normalized per-frame weight tracks that downstream avatar or facial-retargeting systems can ingest. Use `compressLocal3DVideoToMesh4DGSForStreaming(...)` or `Local3DVideoToMesh4DGSStreamingCompressionPipeline` when the trained `FourDGSTrainingArtifact` needs to be shipped as a bandwidth-aware animated Gaussian package. The workflow accepts in-memory or JSON artifacts, writes a compression report, streaming manifest, chunk payload directory, optional bootstrap plan, and optional decode-preview estimate, and supports separate mobile, web, desktop, and archive delivery profiles. Use `editLocal3DVideoToMesh4DGS(...)` or `Local3DVideoToMesh4DGSEditingPipeline` when you need non-destructive animated Gaussian scene modification before streaming or export. The workflow accepts in-memory or JSON artifacts plus a sequence of edit operations, emits a semantic workspace, writes the edited anchor-state manifest, and optionally samples preview frames across time so transform, appearance, motion, duplicate, and hide operations can be inspected locally. Use `fuseLocal3DVideoToMeshMultiFrameDepth(...)` or `Local3DVideoToMeshDepthFusionPipeline` to align the per-frame relative depth maps against COLMAP sparse points and emit a fused `.ply` point cloud plus an optional `.npz` point cache. Use `reconstructLocal3DVideoToMeshSurface(...)` or `Local3DVideoToMeshSurfaceReconstructionPipeline` to convert the fused point cloud into an untextured mesh with Poisson surface reconstruction. Use `projectLocal3DVideoToMeshTexture(...)` or `Local3DVideoToMeshTextureProjectionPipeline` to project source-frame colors onto the reconstructed mesh as vertex colors with raycast visibility checks. Use `segmentLocal3DVideoToMeshObject(...)` or `Local3DVideoToMeshObjectSegmentationPipeline` to isolate a foreground object before reconstruction. The workflow accepts a raw video, a frame manifest, or a frames directory; defaults to SAM 2 small with auto-central prompting; and produces mask PNGs, masked RGB frames, optional RGBA cutouts, and an optional masked preview video. Use `reconstructLocal3DVideoToMeshTurntable(...)` or `Local3DVideoToMeshTurntableReconstructionPipeline` when the object is rotating on a turntable while the camera stays mostly fixed. The workflow can start from a raw video and run inline segmentation, or consume the segmentation manifest from the prior workflow directly. It exports selected-view and synthetic-pose manifests, a carved point cloud, an untextured mesh, and a vertex-colored mesh. Use `assessLocal3DVideoToMeshVideoQuality(...)` or `Local3DVideoToMeshVideoQualityAssessmentPipeline` before reconstruction when you need to detect insufficient viewpoint coverage, motion blur, or poor lighting. The workflow accepts a raw video, a frame manifest, or a segmentation manifest; exports a reusable sampled-frame manifest; and returns coverage, sharpness, lighting, issue, and recommendation summaries. Use `previewLocal3DVideoToMeshProgressiveReconstruction(...)` or `Local3DVideoToMeshProgressiveReconstructionPreviewPipeline` when you want checkpointed previews of the reconstruction forming over time. The workflow can run inline from a raw video or consume prepared depth and pose manifests, emits checkpoint point clouds or meshes plus preview PNGs, and assembles an `.mp4` timeline that clients can display as a progressive reconstruction preview. Use `trackLocal3DVideoToMeshVisualSlam(...)`, `Local3DVideoToMeshVisualSlamPipeline`, or `Local3DVideoToMeshVisualSlamSession` when you need real-time stereo camera tracking plus sparse landmark mapping. The shipped workflow is observation driven: it accepts an in-memory stereo feature sequence or a JSON manifest of stereo observations, triangulates sparse landmarks, estimates per-frame poses, detects place-signature loop closures, supports localization-only replay from a serialized map, and writes reusable trajectory, landmark, loop-closure, map-snapshot, and localization-metrics artifacts for downstream SLAM and Gaussian-conversion tasks. Use `fuseLocal3DVideoToMeshLidar(...)` or `Local3DVideoToMeshLidarFusionPipeline` when capture originates from ARKit LiDAR hardware. The workflow accepts either an in-memory ARKit LiDAR frame sequence or a JSON manifest of frames containing metric `sceneDepth` / `smoothedSceneDepth`, confidence maps, intrinsics, and camera transforms. It outputs processed per-frame metric depth artifacts, a trajectory manifest, a confidence report, and a fused ARKit-world `.ply` point cloud instead of pretending monocular relative depth is already metric. Use `convertLocal3DVideoToMeshSlamToGaussianSplatting(...)` or `Local3DVideoToMeshSlamToGaussianSplattingPipeline` when you need a Gaussian scene directly from the shipped SLAM stack. The workflow accepts a shipped `visual-slam` manifest and can optionally accept a shipped `lidar-fusion` manifest; if LiDAR is present, it aligns the SLAM trajectory into ARKit world coordinates with a similarity fit, exports aligned sparse-map sidecars, and packages the final Gaussian scene as both Graphdeco-style `.ply` and Isis `.isplat` artifacts. Without LiDAR it still exports a sparse Gaussian anchor scene in visual-SLAM coordinates, but the output remains non-metric and the colorization is confidence-coded rather than photometric. Use `scanLocal3DVideoToMeshRoomScale(...)` or `Local3DVideoToMeshRoomScaleScanningPipeline` when you want a full room scan from the shipped SLAM stack instead of only intermediate artifacts. The workflow can run inline from stereo observations plus ARKit LiDAR frames or reuse shipped `visual-slam` and `lidar-fusion` manifests directly; it exports a Poisson room mesh, a metric Gaussian scene, and a room report that estimates room bounds, footprint area, room height and volume, floor/wall/ceiling coverage, traversal completeness, and follow-up capture guidance. Use `scanLocal3DVideoToMeshObjectScale(...)` or `Local3DVideoToMeshObjectScaleScanningPipeline` when the target is a single object captured with an ARKit LiDAR device. The workflow accepts raw stereo + LiDAR sequences or the shipped `visual-slam` and `lidar-fusion` manifests, optionally reuses segmentation framing signals, writes a cleaned foreground `.ply`, reconstructs an object mesh from that filtered cloud, and persists a guidance report describing the exact views still worth capturing before final export. Expected Python-side runtime dependencies: - `torch` - `opencv-python` - `numpy` - official `Depth-Anything-V2` checkout or installed `depth_anything_v2` module for the preferred backend - `transformers` for fallback loading via Hugging Face - `colmap` for camera-pose estimation - `opencv-python` for frame and depth image loading during fusion - `numpy` for sparse-calibrated point-cloud fusion - `open3d` for Poisson surface reconstruction and normal estimation - `opencv-python` for source-frame sampling during texture projection - official `sam2` package for promptable video object segmentation - `open3d` for visual-hull carving, Poisson extraction, and mesh coloring in turntable reconstruction - `opencv-python` and `numpy` for capture-quality metrics and sampled-frame staging in video quality assessment - `open3d`, `opencv-python`, and `numpy` for checkpoint preview rendering and progressive preview video assembly - `@isis/3d-inference-local` with its TRELLIS.2 multi-image runtime dependencies for the neural multi-view reconstruction workflow ## Planned Tracks - `monocular-video-to-3d` - `multi-view-reconstruction` - `4d-gaussian-splatting-to-animated-mesh` ## Supported Tracks - `slam-based-3d-scanning` ## Manifest Use `createLocal3DVideoToMeshPackageManifest()` to inspect the current package status plus the remaining planned `71.4` roadmap surface without overstating support.