LEGO: Leveled Language Gaussian Splatting
Authors: Yuning Peng, Haiping Wang, Yuan Liu, Yipeng Lu, Zhen Dong, Bisheng Yang
Published: Accepted to ECCV 2026; submitted to arXiv on August 10, 2026
Abstract: We introduce LEGO, a novel framework for advanced open-vocabulary 3D scene understanding. Unlike conventional methods that focus solely on basic concept recognition, LEGO's core innovation is its ability to capture intrinsic semantic hierarchies within a sceneβsuch as the lineage "flowerpot β bouquet β bud β petal." While foundation models like SAM can identify multi-granular structures in 2D, their partitions are strictly perspective-dependent and lack cross-view consistency. LEGO self-adaptively re-grades volatile multi-view SAM granularities into a unified, 3D-consistent hierarchy, providing precise supervision for structurally coherent, multi-level 3D segmentation. By grounding these segments with CLIP embeddings, LEGO recovers open-vocabulary semantic logic across hierarchical levels. Furthermore, by incorporating spatial relationships, we elevate these segments into level-wise language scene graphs, effectively empowering Large Language Models to perform complex, context-aware spatial reasoning and precise visual grounding. Experimental results demonstrate that LEGO achieves state-of-the-art performance on both promptable and open-vocabulary 3D segmentation benchmarks, exhibiting advanced hierarchical scene decomposition and context-aware spatial reasoning.
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2608.10057 [cs.CV]
DOI: https://doi.org/10.48550/arXiv.2608.10057
Project Page: LEGO Webpage
1. Introduction
In the rapidly evolving field of 3D vision, open-vocabulary scene understanding has become a critical capability for applications ranging from autonomous navigation to augmented reality. By 2026, the demand for systems that can not only identify objects but also comprehend their semantic relationships and hierarchical structures has intensified. Traditional methods, often reliant on closed-set labels, fall short in dynamic, real-world environments where novel categories and fine-grained distinctions are common.
LEGO addresses this gap by introducing a leveled approach to language-guided Gaussian splatting. It goes beyond recognizing individual concepts, aiming to capture the rich, nested semantic relationships that define how we perceive and interact with the world.
2. Method
LEGO's framework is built on three key pillars:
- Unified Semantic Hierarchy from Multi-View Data: The process begins with generating multi-granular segmentation proposals using 2D foundation models like SAM (Segment Anything Model). However, these proposals are inherently view-dependent and inconsistent across different camera angles. LEGO introduces a self-adaptive re-grading mechanism that consolidates these volatile granularities into a single, coherent 3D-consistent hierarchy. This unified structure provides robust supervision for multi-level segmentation of the 3D scene.
- Open-Vocabulary Semantic Grounding: Each segment in the hierarchical structure is grounded using CLIP embeddings. This grounding enables the model to associate 3D regions with open-vocabulary semantic concepts, moving beyond fixed category sets. By doing so, LEGO can recover semantic logic across different levels of the hierarchy, allowing for both coarse (e.g., "flowerpot") and fine (e.g., "petal") distinctions.
- Level-wise Language Scene Graphs: To further enhance context-aware reasoning, LEGO integrates spatial relationships between segments into a leveled scene graph structure. This graph, enriched with language descriptions, serves as a sophisticated interface for Large Language Models (LLMs). It allows the system to answer complex queries that require spatial and relational understanding, such as "Find the bud that is inside the red flowerpot on the table."
- Higher accuracy in segmenting fine-grained object parts and hierarchies.
- Better cross-view consistency, leading to more stable 3D reconstructions.
- Enhanced performance in complex reasoning tasks involving spatial relationships.
3. Results
We evaluate LEGO on multiple public benchmarks for 3D segmentation, including both promptable (e.g., point-based) and open-vocabulary settings. Our experiments show that LEGO consistently outperforms existing methods, setting new state-of-the-art records. Key improvements include:
4. Conclusion
LEGO represents a significant step forward in 3D scene understanding by combining the strengths of foundation models with a novel hierarchical approach. Its ability to construct level-wise language scene graphs bridges the gap between low-level geometric segmentation and high-level semantic reasoning. As 3D applications continue to grow in complexity, frameworks like LEGO will be instrumental in enabling machines to see and reason about the world with human-like nuance.
This paper is available under the arXiv preprint identifier 2608.10057 and has been accepted for publication at the European Conference on Computer Vision (ECCV) 2026.
via ArXiv CV