Key Takeaways
- The pattern: This is Google's Open Knowledge Format skeleton β a Markdown file with a YAML frontmatter block β repurposed for agent hand-off. The repo's frontmatter carries one extra load-bearing field that the general OKF spec does not define:
token_pointer, an absolute path to the pre-computed.npyarray in shared memory. Human-readable body, machine-readable pointer. - The mechanism: Three Qwen2.5-Coder models of different sizes (7B / 3B / 1.5B) cannot share a KV cache β they have different architectures. But they can share pre-computed token IDs, because the whole Qwen2.5-Coder family ships one identical BPE vocabulary. This repo tokenizes once, hands off the integer array through
/dev/shm/qwen_tokens/, and lets every downstream agent skip its own tokenizer entirely on the input side. - The numbers: Median of 7 trials per prompt, 3 blocks, greedy decoding, 64 new tokens: on the 3B model, mean baseline TTFT drops from 69.3 ms to 49.9 ms β a 28.0% reduction. On the 1.5B model, from 49.6 ms to 30.9 ms β a 37.8% reduction. Both models pass the coherence heuristic on every sample. Full pipeline wall clock is 41.3 s end to end (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).
- The guardrail: Feeding a downstream model an integer array that meant a different subword under its own vocabulary does not crash anything. It generates a fluent, coherent-looking, completely wrong report. So before any agent trusts another agent's integers, this pipeline runs a full ~151,936-entry
get_vocab()dict equality check β not avocab_sizecomparison, the real thing. - What this does NOT claim: Short-block regime (few-hundred-token blocks). No custom CUDA β this is orchestration on top of
transformers' existingmodel.generate(input_ids=...)API. Tokenizer equivalence is verified for the exact three checkpoints this repo pins, not a family-wide standing guarantee.
TL;DR up front, so you can leave with the point: If you have ever wired three or more LLM-based agents from the same model family into a pipeline that fans out over one shared document, your CPU is running the exact same Byte-Pair Encoding merges over the exact same characters two or three times in a row, because each agent's tokenizer is a stateless newborn that has no idea the previous agent already produced the same integer array. This post is about a small pipeline of three Qwen2.5-Coder models (7B, 3B, 1.5B) where the upstream agent tokenizes once, drops a NumPy array of int64 token IDs into /dev/shm/qwen_tokens/, and every downstream agent calls model.generate(input_ids=...) directly on that array. It also β and this is where the actually interesting engineering lives β refuses to let anyone else in the pipeline trust that array until it has confirmed, byte for byte, that every model in the chain agrees on what those integers mean. This is orchestration, not a CUDA kernel. But if you have ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the shape of the problem this piece of infrastructure is designed to prevent.
GitHub repo: https://github.com/AnubhabBanerjee/inter-llm-tokf
1. The Confession: Your Second Agent is Doing Your First Agent's Homework, Twice
Let me dramatize the moment this whole repo is about.
Imagine you have three LLM agents chained together. Agent 1 is a large model that reads a design document. Agent 2 is a mid-sized model that evaluates part of it. Agent 3 is a small model that writes the final report. All three come from the same model family β same tokenizer, same vocabulary, same everything above the hidden layers β just at three different sizes. Since you are not made of H100s, running a 7B model three times when a 1.5B model will do for the last step would be, frankly, rude to your GPU.
Now, here's the catch: when Agent 1 reads the document, it runs its tokenizer and produces a list of integer IDs. Agent 2 then reads the same document β and the same tokenizer runs again, on the same characters, producing the same integers. Agent 3 does it a third time. In any standard setup, that's three calls to the same BPE merge logic on the same input, wasting CPU cycles that could be spent on inference or other tasks. The waste becomes even more pronounced in 2026, as multi-agent pipelines grow in complexity and shared-document fan-out becomes a common pattern for tasks like report generation, code review, and knowledge synthesis.
This repo's approach is straightforward: tokenize once, store the resulting int64 token array in a shared memory location, and let every downstream agent use it directly. But there's a subtle problem: what if the integers don't mean the same thing to every model? That's where the heavier engineering comes in.
2. Beyond Token Sharing: The OKF Skeleton and the token_pointer Field
The Open Knowledge Format (OKF) provides a structured way to package datasets and models with metadata, but it wasn't designed for inter-agent hand-off. This repo borrows the OKF skeleton β a Markdown file with a YAML frontmatter block β and adds a custom field, token_pointer, to point to a pre-computed NumPy array in /dev/shm. This tiny extension makes the format immediately usable for agent-to-agent communication.
Why /dev/shm? It offers shared, memory-mapped storage on Linux systems, allowing fast access by multiple processes without hitting the filesystem. In a 2026 landscape where agent orchestration frameworks (like LangChain and AutoGen) often run multiple models in separate processes, memory-mapped shared arrays are a lightweight, cost-effective alternative to serializing and deserializing token lists or using slower IPC mechanisms.
3. The Guardrail: Vocabulary Equivalence as a Trust Barrier
The critical safety check in this pipeline is a full vocabulary comparison across all models. A naive vocab_size check is never sufficient, because two models could have the same vocab size but different term-to-ID mappings. The repo's guardrail runs a get_vocab() equality check across the three models' tokenizers to ensure the integer arrays are interoperable. Failure to do so would produce 'fluent, coherent-looking, completely wrong' outputs β a subtle failure mode that's all too common in optimization efforts.
As of 2026, with tokenizer optimizations (like byte-level BPE variants) becoming more prevalent, this guardrail is a time-sensitive best practice. It prevents silent data corruption in knowledge exchange pipelines, where a mismatch could cause downstream agents to generate outputs that appear plausible but are fundamentally incorrect.
4. Performance, Scope, and Limitations
The numbers are promising: on the 3B model, TTFT drops by 28% (from 69.3 ms to 49.9 ms); on the 1.5B, it drops by 37.8% (from 49.6 ms to 30.9 ms), with a full pipeline wall clock of 41.3 seconds. However, the author is careful to state the limitations: this work targets short-block regimes (a few hundred tokens), and the tokenizer equivalence is guaranteed only for the exact three checkpoints pinned in the repo, not as a family-wide rule.
For 2026 practitioners, these numbers illustrate a broader trend: reducing redundant tokenization is a low-hanging fruit in pipeline optimization. As models grow in size and complexity, the CPU cost of tokenization becomes a larger fraction of the total latency, especially for prompt-heavy workloads. Sharing token arrays can yield measurable gains, but only when vocabulary alignment is verified.
5. Implementation and Future Directions
From a practical standpoint, the pipeline uses the transformers library's existing model.generate(input_ids=...) API, meaning no custom kernels are needed β it's pure orchestration. This makes it easy to port to other model families that share a single vocabulary (e.g., GPT-4-style models or LLaMA 3 within the same size family). Looking ahead, one could extend the idea to:
- Handling context blocks longer than a few hundred tokens.
- Verifying vocabulary equivalence for a wider range of model sizes (e.g., 7B/13B/30B) to make the guardrail more scalable.
- Integrating with distributed storage systems, such as Redis or Mleap, for cross-node pipelines.
Conclusion
In the age of multi-agent LLM systems, knowledge exchange is becoming a central design problem. This repo demonstrates that reusing token arrays across models from the same family is a viable efficiency boost, provided you put trust-checking first. It's a reminder that meaningful AI infrastructure improvements don't always require GPU wizardry β sometimes they're as simple as saving your CPU a few seconds of repeating itself, with a guardrail to keep things right.