How to Run NVIDIA Dory 1.0 4B with llama.cpp: A Loop-by-Loop Guide

Overview

In this tutorial, we work with Dory 1.0 4B, a small reasoning model from NVIDIA's Nemotron Labs. Its middle layers form a shared block that can be applied once, twice, three, or four times per token. Each additional pass — or loop — spends more computation per token without adding a single parameter, so a single checkpoint can run at four different compute budgets with no retraining. This is the same looped-architecture reasoning trend that has gained traction across small-model releases heading into 2026, where adaptive compute per token is used to trade latency for accuracy on demand.

We will:

1. Build the llama.cpp branch pinned by NVIDIA's model card.
2. Download the 3.7 GB GGUF and read the architecture directly from the file to predict the parameter and cache-memory cost of each loop.
3. Validate those predictions against the running server.
4. Measure decode speed at every loop count.
5. Test whether the cheaper last-loop cache changes the model's predictions.
6. Run a paired GSM8K evaluation to observe how accuracy, answer length, and time per solved question shift as loops increase.



Step 0: Setup and Helpers




Start with the scaffolding used throughout the guide: imports, a results registry for the summary table, a claim-tracking list, and a section banner helper.




import os
import re
import sys
import json
import time
import math
import random
import signal
import shutil
import hashlib
import platform
import statistics
import traceback
import subprocess
import collections
import urllib.request
from pathlib import Path

RESULTS = {}          # step -> one-line result, for the summary table
CLAIMS = []           # (step, claim, held)
CURRENT = ["1"]


def banner(title):
    print("\n" + "=" * 78)
    print(title)
    print("=" * 78)


def section(name):
    def wrap(fn):
        def run(*a, **kw):
            CURRENT[0] = name.split(".")[0]
            banner(name)
            t0 = time.time()
            try:
                out = fn(*a, **kw)




Note: The original snippet continues past this point; the remaining helper definitions were truncated. The scaffolding above is enough to follow the structure of every subsequent section.




Step 1: Build the Pinned llama.cpp Branch




NVIDIA's model card specifies the exact llama.cpp commit required to load Dory 1.0 4B. Clone that branch rather than using a release build, then compile with GGUF support enabled.




git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout <pinned-commit-from-model-card>
cmake -B build -DLLAMA_CURL=ON
cmake --build build --config Release -j




Confirm the build succeeded by checking the server binary before proceeding.




Step 2: Download the GGUF




Pull the 3.7 GB BF16 GGUF from the Hugging Face repository linked in the model card:




huggingface-cli download nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
  --include "*.gguf" --local-dir ./models




Step 3: Read the Architecture from the File




Before running inference, inspect the GGUF metadata to extract block counts, layer counts, and head dimensions. This lets us predict the parameter footprint and KV-cache size for each loop count analytically, then verify against the live server later.




Parse the GGUF header and dump the .blockcount, .attention.headcount, and embedding-size keys into a small table. Because Dory's middle layers are reused across loops rather than duplicated, the parameter count should remain constant across loop settings while the KV-cache and compute cost scale with the loop count.




Step 4: Validate Predictions Against the Running Server




Launch llama-server with the downloaded GGUF, then query the /props endpoint to read back the effective context size and model parameters. Compare these to the values predicted in Step 3. Any mismatch here indicates either a wrong llama.cpp build or an incorrect metadata interpretation, so resolve it before benchmarking.




./build/bin/llama-server -m ./models/dory-1.0-4b-bf16.gguf \
  --host 127.0.0.1 --port 8080 -c 8192




Step 5: Measure Decode Speed Across Loop Counts




For each loop setting (1, 2, 3, 4), send a fixed prompt and record tokens per second during decode. Since every extra loop adds compute per token without adding parameters, expect decode throughput to drop monotonically with loop count while the memory footprint stays roughly flat. Record the results in the RESULTS dict so they appear in the final summary table.




Step 6: Test the Last-Loop Cache




Dory supports a cheaper cache configuration that stores KV state only for the final loop. Compare greedy outputs with and without this optimization on a set of prompts. If the predictions are identical, the cache is safe to enable for production and reduces memory pressure; if they diverge, note the prompts where they differ so users can decide whether the savings justify the risk.




Step 7: Paired GSM8K Evaluation




Run the standard GSM8K test set at each loop count, using the same prompts and seeds so the comparison is paired. For every loop setting, track:



• Accuracy — fraction of correctly solved questions.
• Answer length — average tokens generated per answer.
• Time per solved question — wall-clock time divided by the number of correct answers.



Because the loops trade compute for reasoning depth, expect accuracy to rise with loop count, answer length to grow, and time per solved question to increase as well. The interesting output is the marginal accuracy gain per additional loop, which quantifies how much reasoning depth each pass actually buys.




Step 8: Summary Table




After all sections run, print the RESULTS and CLAIMS registries. The results table shows each step's one-line outcome; the claims table shows which predictions from the architecture analysis held up and which did not. This closes the loop between the metadata-derived cost model and the measured behavior of the running model.




Takeaways



• One checkpoint, four compute budgets. Dory 1.0 4B runs the same weights at 1–4 loops, so you can dial reasoning depth per token without retraining or swapping models.
• Loops cost compute, not memory. Parameter count stays fixed; only compute and (optionally) KV-cache scale with loop count.
• Verify the build and model card first. The pinned llama.cpp commit is required — release builds may fail to load the GGUF.
• The last-loop cache is worth testing. If it doesn't change greedy outputs on your workloads, it's free memory savings.
• Benchmark with paired evaluations. GSM8K with fixed seeds gives you the accuracy-per-loop curve needed to pick the right setting for your application.

via MarkTechPost