LingBot-Map Tutorial: GPU-Aware Inference and Point Cloud Export

This tutorial demonstrates how to build an end-to-end streaming 3D reconstruction pipeline using LingBot-Map. We'll cover configuration, GPU-aware parameter tuning, model setup, inference, and exporting point clouds and other artifacts. By the end, you'll have a complete workflow ready for your own scenes.

Overview

LingBot-Map is a state-of-the-art framework for real-time 3D scene reconstruction from monocular video or image sequences. It leverages streaming attention and long-range trajectory memory to produce accurate camera poses and dense point clouds, even on limited hardware. This tutorial walks through a complete pipeline—from raw input to exportable 3D models—with special attention to GPU memory management for 2026-era hardware.

Configuration

First, we define a configuration dictionary that controls every stage of the pipeline. This includes input settings, model parameters, inference options, and output preferences.

CFG = {
    "scene":        "courthouse",
    "image_folder": None,
    "video_path":   None,
    "fps":          10,
    "max_frames":   None,
    "stride":       None,
    "checkpoint":   "lingbot-map.pt",
    "image_size":   518,
    "patch_size":   14,
    "use_sdpa":     True,
    "mode":         "streaming",
    "num_scale_frames":      None,
    "keyframe_interval":     None,
    "kv_cache_sliding_window": 64,
    "camera_num_iterations": None,
    "offload_to_cpu":        True,
    "window_size":           128,
    "overlap_keyframes":     8,
    "conf_percentile":       55.0,
    "pixel_stride":          2,
    "max_plot_points":       60000,
    "export_ply":            True,
    "export_glb":            False,
    "launch_viser":          False,
    "run_ablation":          False,
    "seed":                  0,
}

Key settings:

• Scene: Identifier for your dataset, used for output naming.
• Input: Provide either image_folder (sequence of frames) or video_path (video file). Leave the other as None.
• FPS & max_frames: Control frame sampling rate and total number of frames processed.
• Checkpoint: Path to the pretrained model weights.
• Model parameters: image_size, patch_size, and use_sdpa (scaled dot-product attention) affect efficiency and accuracy.
• Streaming settings: keyframe_interval, kv_cache_sliding_window, and window_size control memory usage during long sequences.
• Output: Toggle PLY/GLB exports and 3D visualization via Viser.

Setup and Dependencies

Next, we set up the workspace and clone the repository. We'll also install dependencies and download the pretrained checkpoint.

import os, sys, subprocess, glob, json, time, math, shutil, textwrap
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")

WORK = "/content"
REPO = f"{WORK}/lingbot-map"
OUT  = f"{WORK}/lingbot_out"
os.makedirs(OUT, exist_ok=True)

def sh(cmd, check=True):
    """Run a shell command and print output on failure."""
    p = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    if p.returncode != 0 and check:
        print(p.stdout[-3000:])
        print(p.stderr[-3000:])
        raise RuntimeError(f"Command failed: {cmd}")
    return p

cd $WORK
git clone https://github.com/robbyant/lingbot-map.git
cd lingbot-map
pip install -r requirements.txt
wget -O lingbot-map.pt https://huggingface.co/robbyant/lingbot-map/resolve/main/lingbot-map.pt

GPU-Aware Configuration Tuning

To get the best performance, we dynamically adjust inference parameters based on the GPU's VRAM. This is crucial in 2026 as GPU memory sizes vary widely—from 8 GB laptops to 80 GB datacenter cards. We'll write a function that probes VRAM and sets frame limits, iteration counts, and caching parameters accordingly.

def tune_parameters_for_gpu(cfg, vram_gb):
    """Adjust cfg based on available VRAM."""
    if vram_gb >= 24:
        cfg["max_frames"] = cfg.get("max_frames") or 300
        cfg["camera_num_iterations"] = 200
        cfg["num_scale_frames"] = 8
        cfg["keyframe_interval"] = 10
    elif vram_gb >= 12:
        cfg["max_frames"] = cfg.get("max_frames") or 200
        cfg["camera_num_iterations"] = 150
        cfg["num_scale_frames"] = 5
        cfg["keyframe_interval"] = 8
    else:
        # Low VRAM: reduce load
        cfg["max_frames"] = cfg.get("max_frames") or 100
        cfg["camera_num_iterations"] = 100
        cfg["num_scale_frames"] = 3
        cfg["keyframe_interval"] = 5
        cfg["offload_to_cpu"] = True
        cfg["kv_cache_sliding_window"] = 32  # smaller cache
    return cfg

# Detect VRAM (in GB)
try:
    import torch
    if torch.cuda.is_available():
        vram_gb = torch.cuda.get_device_properties(0).total_memory / 1e9
    else:
        vram_gb = 0
except:
    vram_gb = 0

print(f"Detected VRAM: {vram_gb:.1f} GB")
CFG = tune_parameters_for_gpu(CFG, vram_gb)

Input Preprocessing

We now load frames from either an image folder or a video file, apply center cropping and resizing, and prepare them for the model.

import cv2
import numpy as np

def load_frames(cfg):
    frames = []
    if cfg["image_folder"]:
        paths = sorted(glob.glob(os.path.join(cfg["image_folder"], "*.jpg")))[:cfg["max_frames"]]
        for p in paths:
            img = cv2.imread(p)
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
            frames.append(img)
    elif cfg["video_path"]:
        cap = cv2.VideoCapture(cfg["video_path"])
        count = 0
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            if count % int(cfg["fps"] // 10) == 0 and len(frames) < cfg["max_frames"]:
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                frames.append(frame)
            count += 1
        cap.release()
    # Preprocess: resize to image_size with padding
    processed = []
    for frame in frames:
        h, w = frame.shape[:2]
        scale = cfg["image_size"] / min(h, w)
        new_h, new_w = int(h * scale), int(w * scale)
        resized = cv2.resize(frame, (new_w, new_h))
        # Center crop to image_size
        top = (new_h - cfg["image_size"]) // 2
        left = (new_w - cfg["image_size"]) // 2
        cropped = resized[top:top+cfg["image_size"], left:left+cfg["image_size"]]
        processed.append(cropped)
    return np.array(processed)

frames = load_frames(CFG)
print(f"Loaded {len(frames)} frames")

Model Construction

We instantiate the GCTStream model with streaming attention and trajectory memory. The model also loads the pretrained checkpoint.

from lingbot_map.models import GCTStream

model = GCTStream(
    image_size=CFG["image_size"],
    patch_size=CFG["patch_size"],
    use_sdpa=CFG["use_sdpa"],
    mode=CFG["mode"],
    num_scale_frames=CFG["num_scale_frames"],
    keyframe_interval=CFG["keyframe_interval"],
    kv_cache_sliding_window=CFG["kv_cache_sliding_window"],
    camera_num_iterations=CFG["camera_num_iterations"],
    offload_to_cpu=CFG["offload_to_cpu"],
    window_size=CFG["window_size"],
    overlap_keyframes=CFG["overlap_keyframes"],
)

checkpoint = torch.load(CFG["checkpoint"], map_location="cpu")
model.load_state_dict(checkpoint["model_state_dict"])
model.cuda().eval()

Inference and Point Cloud Generation

We run mixed-precision inference to predict camera poses and depth maps, then convert depth to world-coordinate point clouds.

from lingbot_map.utils.geometry import depth_to_pointcloud, pose_from_parameters

with torch.no_grad(), torch.autocast(device_type="cuda", dtype=torch.float16):
    for i in range(0, len(frames), CFG["pixel_stride"]):
        # Prepare batch of frames
        batch = torch.tensor(frames[i:i+CFG["pixel_stride"]], device="cuda").float() / 255.0
        # Model forward pass
        predictions = model(batch)
        # Extract camera parameters and depth from predictions
        camera_params = predictions["camera_params"]
        depth_map = predictions["depth"]
        # Compute world coordinates
        pts = depth_to_pointcloud(depth_map, camera_params, CFG["conf_percentile"])
        # Accumulate points
        all_points.append(pts.cpu().numpy())

Exporting Results

Finally, we export the 3D reconstruction to PLY (point cloud) and optionally GLB (mesh). We also validate the geometry and optionally visualize in Viser.

from lingbot_map.utils.export import export_ply, export_glb

# Combine all points
all_points = np.concatenate(all_points, axis=0)

# Downsample for visualization/export (optional)
if len(all_points) > CFG["max_plot_points"]:
    indices = np.random.choice(len(all_points), CFG["max_plot_points"], replace=False)
    all_points = all_points[indices]

if CFG["export_ply"]:
    export_ply(all_points, os.path.join(OUT, "scene.ply"))
    print("Exported PLY.")

if CFG["export_glb"]:
    export_glb(all_points, os.path.join(OUT, "scene.glb"))
    print("Exported GLB.")

Conclusion

You now have a complete GPU-aware inference pipeline with LingBot-Map. The framework adapts to your hardware, streams long sequences efficiently, and exports point clouds ready for downstream tasks. This tutorial aligns with 2026 best practices, including mixed-precision computation and dynamic VRAM tuning, ensuring your reconstruction tasks run smoothly across a wide range of hardware.

via MarkTechPost