Last Time on a Smaller Stage
The Setup: A Replay, Not a Re-Run
I pulled 27 real historical task prompts directly from Jarvis’s Langfuse traces (90-day window), stratified across 7 categories — calendar, code, email, files, general, messaging, notes — with 4 each, except one category had 3 due to availability.
Claude’s side of the comparison is its actual historical production answer, already logged in Langfuse. I did not re-run it. Re-running Claude through a sandboxed harness would mean feeding it fake stubbed tool data it never actually saw, handicapping it for no good reason. The honest baseline is the answer it really gave, with the real data it really had. That baseline is frozen: identical in both rounds below. Only the local model is re-run.
The local side is a fresh run through a sandboxed replay harness: the real Jarvis LangGraph agent code, in-process, with every write-capable tool (send email, write to calendar, post to Discord/WhatsApp, write files) intercepted so nothing real happens. Read-only tools that touch real external systems — Outlook COM for email and calendar — are intercepted too, but the mock serves the real recorded output from that task’s original Langfuse trace when one exists, not a generic stub. The local model reasons over the same real inbox and calendar content Claude saw, not invented placeholder text.
The wrapper is default-deny: any tool not on a short explicit allowlist gets mocked. That detail earned its keep — between the two rounds, Jarvis grew new tools, and default-deny meant they were intercepted automatically instead of quietly executing against my real accounts.
Quality is scored independently — not pairwise, to avoid position bias — by an LLM judge (claude-opus-4-8) on a 1–5 scale mapped to 0–100, applied identically to every answer in both rounds.
Worth flagging plainly: the judge is a Claude model scoring Claude’s own answers alongside the local ones. That’s self-preference bias — a documented effect in LLM-as-judge setups, where a model tends to rate outputs from its own family more favorably. I can’t cleanly rule it out, and it probably flatters Claude. It’s a real limitation of this methodology, not a footnote to bury — and it matters more in round 2, where the narrower gap makes even a small bias potentially decisive.
Round 1: A Single RTX 3090, 16K Context
With a 30B model (Qwen3-Coder) on a single RTX 3090 and a 16K context window, the results were sobering. The local model scored 22.8/100 against Claude’s frozen 89.4. More telling than the score: 7 of 27 answers (26%) contained malformed tool-call syntax — the model attempted to invoke tools but produced JSON that failed to parse or execute. These weren’t subtle failures; they were system-breaking errors that would have crashed any production agent.
This round confirmed what many practitioners suspect: small local models struggle with complex tool use, especially under tight context constraints. The 16K context forced aggressive truncation of conversation history and tool outputs, crippling the model’s ability to maintain coherence across multi-step tasks.
Round 2: Three RTX 3090s, 256K Context
After upgrading to three RTX 3090s (via NVLink) and expanding the context window to 256K, I re-ran the same 27 tasks with a 122B model (Qwen3-122B-A3B, a Mixture-of-Experts architecture). The improvement was dramatic: score jumped to 80.0, and malformed tool calls dropped to zero (0 of 27). Equally striking was the cost: $0.000969 per task in electricity versus Claude’s $0.763 — a 787× cost reduction.
Why such a leap? Three factors likely combined: (1) the larger model handles complex tool-use grammar far more reliably, (2) the 256K context allows full conversation and tool-output retention without truncation, and (3) the MoE architecture activates only relevant parameters per token, keeping inference efficient even at 122B scale.
My agent now runs on this local setup. I cancelled Claude Max and kept Pro as a fallback for tasks where the local model still underperforms (e.g., nuanced creative writing).
The Honest Caveats
I must stress the confounds: model size and context budget changed together, so I can’t attribute the improvement to one variable. I ran no control (e.g., same model with larger context, or a different model with same hardware). Future work should isolate these factors.
Additionally, the self-preference bias in the judge likely inflates Claude’s scores, especially in round 2. A blind test with a non-Claude judge (e.g., GPT-5 or a human evaluator) would strengthen confidence.
Finally, this is a system test, not a model test. The harness, tool wrappers, and replay logic are part of what changed. Your mileage will vary with different tools, prompts, and hardware.
Looking Ahead to 2026
As of 2026, local LLM hardware has matured significantly. Consumer GPUs like the RTX 5090 offer 32GB VRAM, and multi-GPU setups with NVLink or PCIe 5.0 are more accessible. Models like Qwen3-122B-A3B demonstrate that MoE architectures can deliver frontier-level performance on modest hardware, and context windows beyond 256K are becoming standard. The cost advantage of local inference will only grow as cloud API prices remain stable or increase with demand.
My experience suggests a practical threshold: for agents with <50 tools and moderate context needs, a mid-range local model (30B–70B) may suffice; for complex, tool-rich agents, aim for 100B+ MoE models with ample context. The gap is closing fast.
Conclusion
Can a local LLM run my AI assistant? Yes — but only with the right hardware and model, and with eyes open about the trade-offs. A single 3090 couldn’t cut it; three could. The 787× cost reduction is real, but so is the 9.4-point quality gap (80.0 vs 89.4). For my workloads, that gap is acceptable. For yours, it might not be.
The honest summary: local LLMs are no longer a toy. They’re a viable, cost-effective alternative for production agents — provided you’re willing to invest in hardware and accept some quality loss. As 2026 unfolds, that line will keep moving in favor of local.