Understanding the Two Layers: Containers vs. Quantization Methods
Most confusion around LLM model formats stems from conflating two distinct layers:
- Container: Defines how tensors are stored on disk (e.g., safetensors, GGUF, PyTorch pickle
.bin/.pt). - Quantization method: Defines how weights are compressed into fewer bits (e.g., GPTQ, AWQ, bitsandbytes NF4, llama.cpp K-quants and I-quants).
- Combined: EXL2 and EXL3 tie a quantization method to a specific storage layout and inference library.
Keeping these layers separate is essential for choosing the right format for your deployment.
Quick Memory Rule of Thumb
Weight memory β parameters Γ bits-per-weight Γ· 8.
| Model | 16-bit | ~4.5 bits per weight |
|---|---|---|
| 8B | ~16 GB | ~4.5 GB |
| 70B | ~140 GB | ~39 GB |
This is arithmetic, not a vendor benchmark. It covers weights onlyβKV cache and runtime overhead add more on top.
1. Full Precision: safetensors and PyTorch .bin
Unquantized models typically ship as 16-bit weights, either as pytorch_model.bin or model.safetensors.
The older .bin/.pt files use Python pickle. Loading a pickle file can execute arbitrary code, making untrusted checkpoints a security risk.
Safetensors, developed at Hugging Face, removes this risk by storing tensors in a simple binary format that cannot execute code. It has become the default for sharing models on the Hugging Face Hub and is widely adopted across the ecosystem in 2026.
2. GGUF: The Universal Container for llama.cpp and Ollama
GGUF (GPT-Generated Unified Format) is a container format designed for llama.cpp and adopted by tools like Ollama, LM Studio, and llamafile. It supports a wide range of quantization methods, including K-quants and I-quants, and is optimized for CPU and Apple Silicon inference.
In 2026, GGUF remains the go-to format for local, cross-platform deployment, especially on consumer hardware. Its single-file design bundles model weights, metadata, and tokenizer configuration, simplifying distribution.
3. GPTQ: Post-Training Quantization for GPU Inference
GPTQ (Generative Pre-trained Transformer Quantization) is a post-training quantization method that compresses weights to 4-bit or 3-bit with minimal accuracy loss. It is primarily used with GPU inference libraries such as AutoGPTQ and vLLM.
GPTQ models are often stored in safetensors containers. While it was once the dominant GPU quantization method, newer approaches like AWQ and EXL2 have gained traction for specific use cases.
4. AWQ: Activation-Aware Weight Quantization
AWQ (Activation-aware Weight Quantization) protects the most salient weights based on activation distributions, often yielding better accuracy than GPTQ at the same bit width. It is supported by llm-awq, vLLM, and TensorRT-LLM.
By 2026, AWQ has become a popular choice for production GPU inference, balancing speed and accuracy. Like GPTQ, AWQ models are typically stored in safetensors containers.
5. EXL2 and EXL3: ExLlama's Integrated Format
EXL2 and its successor EXL3 are quantization methods tied to the ExLlamaV2 and ExLlamaV3 libraries. They offer fine-grained control over bits-per-weight and are optimized for NVIDIA GPUs.
EXL2/EXL3 models are stored in a custom layout that includes quantization parameters, making them incompatible with other inference engines without conversion. They are favored by users who want maximum performance from ExLlama's highly optimized kernels.
Choosing the Right Format in 2026
The optimal choice depends on your hardware and use case:
- CPU or Apple Silicon: GGUF (with K-quants or I-quants) is the standard.
- NVIDIA GPU (general): AWQ or GPTQ in safetensors, supported by vLLM and TensorRT-LLM.
- NVIDIA GPU (max performance): EXL2/EXL3 with ExLlamaV2/V3.
- Full precision: safetensors for safety and compatibility.
As the ecosystem evolves, new formats and methods will continue to emerge. Understanding the container vs. method distinction will help you adapt to whatever comes next.
via MarkTechPost