Abstract
This study explores a governed framework for enterprise analytics, in which a language model is used to interpret user queries, while a deterministic policy component selects and executes pre-approved analytical programs that return both results and supporting evidence. We demonstrate that this constrained approach remains sufficiently expressive within a well-defined analytical class, supporting relational operations, aggregation, comparison, window functions, ranking, and similarity queries. The combination of fixed semantics, enforced policy, consistent data, and deterministic execution rules ensures that results are fully replayable. In an empirical evaluation spanning 440 runs, three different 8B-parameter models generated SQL and selected tools at runtime. In contrast, Qwen3-8B was used solely for intent interpretation, with policy driving execution of the approved program. Notably, none of the 330 runtime-planning episodes satisfied the complete answer-and-evidence contract across all test datasets, whereas the policy-executed analyzer succeeded in all 110 of its corresponding runs. These findings are specific to the configurations tested and do not preclude the possibility that runtime agents might succeed under alternative designs.
Introduction
Large language models (LLMs) are increasingly considered for natural language interfaces to enterprise data systems. However, unrestricted runtime planning—where the model generates both the analytical approach and the code—poses significant risks around interpretability, governance, and compliance. In response, we propose a hybrid architecture that separates intent interpretation from action selection. This paper investigates whether such a governed system can maintain analytical expressiveness while ensuring reliability and replayability.
Methodology
We adopted a two-tier architecture. In the first tier, an instruction-tuned LLM (Qwen3-8B) or a general-purpose LLM (three variants of 8B size) interprets the user's natural language question. In the second tier, a deterministic policy selects a pre-approved analytical program based on the interpreted intent. This program is executed against the enterprise data store, producing both the query result and audit-friendly evidence of how the result was derived. All data, policies, and execution rules were held constant across runs to guarantee reproducibility.
The supported analytical class included:
- Relational operations (e.g., joins, filters, projections)
- Aggregation (e.g., sum, average, count)
- Comparisons and thresholding
- Window functions (e.g., running totals, moving averages)
- Ranking
- Similarity-based queries
Evaluation
We conducted a total of 440 controlled runs across multiple enterprise-style datasets. For runtime-planning configurations, three 8B-sized LLMs generated SQL and selected tools dynamically. For the governed configuration, Qwen3-8B performed intent interpretation only, and the policy module selected and executed the approved program. Performance was measured by whether each run produced the complete answer-and-evidence contract.
Results
- Runtime-planning configurations: 0 out of 330 runs fully met the contract.
- Policy-executed configuration: 110 out of 110 runs fully met the contract.
The results indicate that, under the tested settings, runtime planning by LLMs failed to meet the stringent evidence requirements of enterprise analytics. In contrast, policy-based execution achieved perfect compliance and consistency.
Discussion
The failure of runtime-planning models to meet the full contract likely stems from their tendency to deviate from approved analytical procedures or to lack adequate self-verification. However, these results are configuration-specific; other model sizes, prompts, or training regimes might yield different outcomes. The strength of the governed approach lies in its enforcement of a deterministic policy and its emphasis on evidence generation, which aligns with emerging industry requirements for AI transparency and auditability.
As of 2026, enterprises are increasingly adopting hybrid human-AI decision frameworks, and regulatory pressures continue to rise. Systems that can demonstrate controlled execution and replayability thus have a significant advantage in producing compliant and trustworthy analytical outputs.
Conclusion
A governed, policy-driven approach to analytical query execution can achieve high reliability and accountability without sacrificing expressiveness within a constrained operational scope. By separating intent interpretation from action selection and execution, we enable the benefits of LLM-based interfaces while mitigating risks of nondeterminism and ungoverned code generation. This work contributes to the ongoing effort to integrate AI capabilities into enterprise environments under safe and controlled conditions.
via ArXiv AI