BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming
Authors: Clemens Kujat, Bharathi Kannan Nithyanantham, Stefan Telgmann, Tobias Sesterhenn, Joern Ploennigs, Stefan Lüdtke, Christian Bartelt
arXiv: 2611.09933 [cs.CL] · Submitted 13 Oct 2026
Journal reference: Proceedings of the European Conference on Computing in Construction, 2026, pp. 113–120
DOI: https://doi.org/10.48550/arXiv.2611.09933
Overview
Large Language Models (LLMs) are increasingly emerging as natural-language interfaces for Building Information Modeling (BIM). While they have demonstrated success in information retrieval tasks, no benchmark currently exists to evaluate their ability to edit Industry Foundation Classes (IFC) models—the open, vendor-neutral data format that underpins BIM interoperability across the AEC (architecture, engineering, and construction) industry.
The BIBIMBAP Benchmark
To address this gap, the authors introduce BIBIMBAP, a Text-to-BIM benchmark comprising 100 curated tasks that cover Create-Read-Update-Delete (CRUD) operations across five categories:
- Spatial
- Geometric
- Topological
- Numeric
- Conceptual
Each task is packaged with:
- A natural-language prompt
- An associated IFC model
- Expected structured outputs
- Test scripts for automated evaluation
- Constraint preservation — models frequently violate or overlook constraints encoded in the BIM model.
- Spatial reasoning — models struggle to reason accurately about spatial relationships between building elements.
This design enables reproducible, end-to-end assessment of how well LLMs can translate instructional language into correct, constraint-respecting IFC edits.
Baseline Findings
Baseline evaluations reveal limited performance, with the best model achieving only 50.2%. Notably, failures cluster around two recurring weaknesses:
An additional and instructive observation is that some models produce correct outputs despite flawed intermediate reasoning—a pattern that underscores the importance of structured, verifiable evaluation rather than reliance on surface-level plausibility.
Significance
As LLM-driven BIM interfaces move toward practical deployment in 2026 and beyond, BIBIMBAP provides a much-needed standardized yardstick for measuring editing capability—shifting the conversation from retrieval assistance toward reliable, automated model authoring. The benchmark is positioned to help researchers and practitioners identify where current LLMs fall short and to guide the development of more robust BIM-oriented language agents.
Subjects: Computation and Language (cs.CL)
via ArXiv CL+LG