Running Underdog Saluki 27B: A 2-Bit Qwen3.8-27B Tool-Calling Guide

Overview

This tutorial walks through Underdog Saluki 27B, a 7.9 GB build of Qwen3.8-27B. According to its creators, it preserves tool-calling capability at roughly two bits per weight, and on their own 120-task benchmark it reportedly edges out the full 54 GB model. Saluki is built on ISTA-DASLab's GSQ-RCO quantization of the same base model, so we evaluate both files side by side in stock llama.cpp.

The workflow proceeds in stages:

Inspect the files — a tensor-by-tensor diff of the two GGUF headers, followed by a byte-level comparison to determine whether Saluki is a remix of ISTA's bits or a genuinely modified model.
Serve each file on a single GPU — measuring memory footprint and throughput.
Score both models — on the same seeded 120 tasks from the Berkeley Function Calling Leaderboard (BFCL), using BFCL's own checker with strict and lenient parsing.

Getting Started: Imports and Harness Setup

The harness collects per-step results in RESULTS for a summary table and records claim verifications in CLAIMS as (step, claim, held) rows, where held may be True, False, or None (not decided in this run). The CURRENT list tracks the active step for reporting.

import os
import re
import ast
import sys
import json
import math
import copy
import enum
import glob
import time
import types
import random
import shutil
import struct
import hashlib
import tarfile
import platform
import traceback
import subprocess
import collections
import urllib.request
import urllib.error

RESULTS = {}   # step -> one-line result, for the summary table
CLAIMS = []    # (step, claim, held) rows; held is True, False or None (not decided in this run)
CURRENT = ["1"]


def banner(title):
    print("\n" + "=" * 78)
    print(title)
    print("=" * 78)


def section(name):
    def wrap(fn):
        def run(*a, **kw):
            CURRENT[0] = name.split(".")[0]
            banner(name)
            t0 = time.time()
            try:
                out = fn(*a, **kw)
                RESULTS[name] = out if isinstance(out, str) else "ok"
                return out
            except Exception as e:
                RESULTS[name] = f"SKIPPED / FAILED -> {type(e).__name__}: {e}"
                print(f"\n[!] {name} did not complete: {type(e).__name__}: {e}")
        return run
    return wrap

Note: The shell required to reproduce this run (build, llama-server, BFCL's ast_checker) is assumed to be already available. See the original walkthrough for the remaining helper functions and the full end-to-end script.

Why This Matters in 2026

Low-bit quantization has matured considerably: in 2026, 2-bit GGUFs that retain structured capabilities like tool calling are increasingly practical for single-GPU deployments, where a 54 GB full-precision model would otherwise require multi-GPU or high-VRAM setups. This guide's methodology — header diffing, byte-level provenance checks, and BFCL scoring — provides a reproducible way to verify aggressive quantization claims rather than take them on faith.

via MarkTechPost

Running Underdog Saluki 27B: A 2-Bit Qwen3.8-27B Tool-Calling Guide