Abstract
Coding agents allocate a significant portion of their context budget to retrieval operations. Lexical retrieval (grep) is universal, immediate, and requires no setup, but it is noisy: it cannot distinguish a definition from a call or a comment. Semantic retrieval via the Language Server Protocol (LSP) is precise and type-aware, yet it demands a running, indexed server and incurs per-symbol round-trip costs. The assertion that semantic retrieval is more token-efficient is widely claimed but rarely measured; no public source isolates the LSP-vs-lexical token delta for an agent at equal task success.
This paper formalizes the question with a single metric—tokens-to-success—and specifies a five-arm ablation that isolates semantic retrieval from confounding factors. We map three pre-stated failure modes onto measurable variables and report a preliminary study across Python and TypeScript repositories, evaluating Claude Opus 4.8, Sonnet 4.6, and Haiku 4.5. The answer is conditional and often negative. On symbol-named localization tasks, the LSP increases token usage (+6% to +118%), and agents tend to ignore it when freely available. For reference-completeness, the LSP improves precision but does not reduce token consumption, nor can it raise the recall ceiling set by agent thoroughness; token savings appear only for the weakest model.
Tool selection is task-dependent: models default to grep for localization (using semantic retrieval only 0–6% of the time) but spontaneously choose the LSP about half the time on reference tasks. The gap becomes starkest in edit scenarios scored by real test execution. Grep handles multi-file renames perfectly, while a location-only LSP fails three-quarters of these tasks by missing call sites. Even a complete, index-warmed, text-enriched LSP—which includes each reference's line inline, as production LSP-MCP servers do—recovers most but not all of the gap, since renames must also touch comments and strings that semantic references exclude. The implication is not that LSP should always be used, but that an adaptive router tuned to task class, model capability, and lexical noise is necessary.
1. Introduction
In the rapidly evolving landscape of AI-assisted software development, coding agents rely heavily on retrieving relevant code context to perform tasks. However, the context window—the token budget available to the model—is finite, and retrieval strategies directly impact efficiency and success. As of 2026, with models like Claude Opus 4.8 and Sonnet 4.6 setting new benchmarks, optimizing retrieval remains a critical bottleneck.
Two dominant retrieval paradigms exist: lexical retrieval (grep) and semantic retrieval (via LSP). Lexical retrieval is simple and universal, but it returns raw text matches without syntactic or semantic understanding. Semantic retrieval, by contrast, provides structured, typed information about symbols, references, and definitions, but requires an active language server and incurs latency and token costs per request.
Despite widespread claims that semantic retrieval is more token-efficient, empirical evidence is scarce. This paper addresses this gap by introducing a rigorous measurement methodology and presenting a preliminary study that compares LSP and grep across various tasks and model architectures.
2. Methodology
We define "tokens-to-success" as the primary metric: the number of tokens consumed by an agent to complete a task with verified success. This metric allows for a fair comparison across retrieval strategies, independent of model or task complexity.
To isolate the effect of semantic retrieval, we design a five-arm ablation study:
- Grep-only: Baseline using lexical retrieval exclusively.
- LSP-only (location): Semantic retrieval providing only symbol locations.
- LSP-only (reference): Semantic retrieval providing reference lists without inline text.
- LSP-enriched: Semantic retrieval with inline text for each reference, simulating production LSP-MCP servers.
- Adaptive: A router that chooses between grep and LSP based on task type and model capability.
- False negatives: Missed symbols due to lexical noise (e.g., comments or strings containing keywords).
- False positives: Irrelevant matches that consume tokens without aiding task completion.
- Context overrun: Excessive token usage due to verbose retrieval results, reducing available context for reasoning.
- Task type: Classification into localization, reference, or edit categories.
- Model capability: Weaker models may benefit from the LSP's noise reduction, while stronger models can handle grep's noise.
- Lexical noise: If identifiers are likely to appear in comments or strings, grep may be preferable for edits; otherwise, LSP for precision.
We pre-state three failure modes and map them to measurable variables:
3. Experimental Setup
We conduct experiments on two repository types—Python and TypeScript—using three Claude models: Opus 4.8, Sonnet 4.6, and Haiku 4.5. Tasks include symbol-named localization, reference-completeness checks, and multi-file edits (e.g., renames) scored by real test execution.
All code and data are publicly available at https://github.com/Poytr1/lsp-vs-grep-token-study.
4. Results
4.1 Symbol-Named Localization
On tasks requiring locating a symbol by name (e.g., finding a function definition), the LSP consistently costs more tokens than grep, with increases ranging from +6% to +118% depending on the model. Notably, when both tools are available, agents often ignore the LSP, using it in only 0–6% of cases, suggesting that the overhead outweighs perceived benefits for simple lookups.
4.2 Reference-Completeness
For tasks requiring all references to a symbol, the LSP improves precision by filtering out non-code matches, but this does not translate into token savings. The recall ceiling remains limited by the agent's thoroughness; if the agent does not enumerate all references, the LSP cannot compensate. Token savings are observed only for Haiku 4.5, the weakest model tested, which benefits from reduced noise in its limited context window.
4.3 Multi-File Edits
In edit tasks scored by test execution, the starkest differences emerge. Grep solves multi-file renames perfectly because it can modify all matches, including those in comments and strings. A location-only LSP fails three-quarters of these tasks by missing call sites. Even a complete, index-warmed, text-enriched LSP—where each reference's line is included inline—recovers most of the gap but cannot close it entirely. Semantic references deliberately exclude comments and strings, which are essential for a correct rename, leading to incomplete edits.
5. Discussion: Implications for Agent Design
The findings challenge the notion that semantic retrieval is universally superior. Instead, the optimal strategy depends on task class, model capability, and lexical noise. For simple symbol localization, grep is sufficient and more efficient. For reference-heavy tasks, the LSP adds precision but not token efficiency. For edits requiring global consistency, neither approach alone is perfect; grep handles lexical matches, while the LSP excels at identifying semantic references.
We propose an adaptive router that selects retrieval methods based on:
This router could significantly reduce token consumption without sacrificing task success, as demonstrated by the conditional gains observed in our study.
6. Conclusion and Future Work
We have presented the first systematic measurement of token efficiency for LSP versus grep in coding agents, revealing that semantic retrieval is not a panacea. Our methodology, with the tokens-to-success metric and five-arm ablation, provides a reusable framework for evaluating retrieval strategies as language models and tooling evolve. In 2026, as context windows grow and LSP implementations mature, the trade-offs may shift; however, the need for adaptive, task-aware retrieval is likely to persist.
Future work will expand the study to more languages, diverse task types, and additional models, and will explore the integration of learned routers that optimize retrieval in real time. We also plan to investigate the impact of token savings on downstream reasoning quality, beyond task completion.
Acknowledgments
We thank the open-source community for the repositories used in this study and the developers of LSP-MCP servers for their production-grade implementations.
Data and Code Availability
All experiments, code, and data are available at the provided GitHub repository to facilitate replication and further research.
References
[References would be listed here per arXiv style, with DOIs where applicable.]
via ArXiv CL+LG