Model Quantization in Machine Learning

The Memory Wall Problem

ML inference is overwhelmingly memory-bandwidth bound. A GPU's arithmetic units process data faster than memory can deliver it. For Llama-2 7B in FP16, loading ~14 GB of weights on an A100 (2 TB/s bandwidth) takes ~7ms — but matrix multiplications need only ~2ms. The model spends >70% of time waiting for memory.

Quantization attacks this directly. Halving precision from FP16 to INT8 halves bytes transferred, nearly doubling effective bandwidth. INT4 quadruples it. This yields near-linear speedups: 4-bit models run ~3-4x faster than FP16.

The Cost Crisis

• Llama-2 70B FP16: 140 GB → 2x A100-80GB (~₹30 lakh/month)
• Llama-2 70B INT4: 35 GB → 1x A100-80GB (₹15 lakh/month)
• Llama-2 7B INT4: 4 GB → T4 GPU (₹50K/month) or a MacBook

For companies serving millions of users, even 2x cost reduction translates to crores annually.

Edge Deployment

On-device ML needs strict constraints — a smartphone has 6-12 GB shared RAM. A FP32 BERT (440 MB) is impractical; INT8 BERT (~110 MB) fits. For on-device LLMs, 4-bit GGUF is the only path.

Why Not Smaller Models?

A 4-bit Llama-2 13B outperforms FP16 Llama-2 7B on most benchmarks despite similar memory. Quantization preserves larger architecture's knowledge capacity — strictly more efficient than training smaller.

Rounding in Everyday Life

A cartographer's GPS data has nanometer precision, but a 1:50,000 map needs only 2 decimal places. Printing more adds zero information — the ink blot exceeds the precision. Similarly, a weight like 0.23847291 in FP32 encodes 23 significant bits, but 0.24 (INT8 mapping) produces effectively the same output. The network never relied on those last 20 bits.

Signal vs Noise

Trained networks have structured weight distributions — bell curves clustered around zero with rare important outliers:

1. Most weights are small and clustered → mapped to few INT8 values with negligible error
2. Outliers are rare but important → AWQ and SmoothQuant handle these specially
3. Redundancy across channels → per-channel quantization adapts to each channel's range

Quantization error isn't uniformly harmful. Errors in small, common weights cancel out across thousands of operations (like noise averaging to zero). Errors in large outliers are disproportionately damaging — hence modern methods focus on protecting outliers.

Precision-Capacity Tradeoff

Measuring a room with millimeter (FP32) vs centimeter (INT8) vs decimeter (INT4) precision: for room-scale objects, centimeters suffice. Resolution loss only matters for sub-centimeter accuracy — analogous to tasks needing fine-grained distinctions. For 95%+ of inference scenarios, 8-bit or 4-bit preserves effective decision boundaries.

Uniform Affine Quantization

Map float $x$ to integer $x_q$ using scale $s$ and zero-point $z$:

$x_q = \text{clamp}\left(\left\lfloor \frac{x}{s} \right\rceil + z, \; q_{\min}, \; q_{\max}\right)$

For $b$-bit: unsigned $q_{\min}=0, q_{\max}=2^b-1$; signed $q_{\min}=-2^{b-1}, q_{\max}=2^{b-1}-1$.

Scale and zero-point from observed min/max:

$s = \frac{x_{\max} - x_{\min}}{q_{\max} - q_{\min}}, \quad z = q_{\min} - \left\lfloor \frac{x_{\min}}{s} \right\rceil$

Dequantization: $\hat{x} = s \cdot (x_q - z)$

Symmetric vs Asymmetric

Symmetric: $z=0$, range $[-\alpha, \alpha]$ where $\alpha = \max(|x_{\min}|, |x_{\max}|)$. Simpler (no zero-point arithmetic) but wastes range for asymmetric distributions (e.g., ReLU outputs).

Asymmetric: non-zero $z$ shifts the range, fully utilizing all $2^b$ levels. More accurate for skewed distributions but adds overhead.

Per-Tensor vs Per-Channel

Per-tensor: one $(s, z)$ for the entire tensor. Simple but loses precision when channels have different ranges.

Per-channel: one $(s, z)$ per output channel. Now standard for weight quantization.

QAT with Straight-Through Estimator

Forward pass simulates quantization; backward pass uses STE since rounding is non-differentiable:

$\frac{\partial \mathcal{L}}{\partial W} \approx \frac{\partial \mathcal{L}}{\partial \hat{W}} \cdot \mathbf{1}_{q_{\min} \leq W/s \leq q_{\max}}$

NormalFloat4 (NF4)

Optimal 4-bit type for normally distributed weights. Levels at quantiles of $\mathcal{N}(0,1)$:

$q_i = \Phi^{-1}\left(\frac{2i+1}{32}\right), \quad i = 0, \ldots, 15$

Minimizes quantization error for normally distributed data, closely matching empirical weight distributions.

What We Covered

Model quantization reduces precision from FP32/FP16 to INT8/INT4, achieving 2-8x memory reduction and 2-4x speedup. The core operation: $x_q = \text{clamp}(\lfloor x/s \rceil + z)$ with scale $s$ and zero-point $z$.

Dominant approaches: GPTQ (Hessian-based optimal weights), AWQ (activation-aware channel protection), bitsandbytes NF4 (optimal 4-bit format), GGUF (CPU/edge format). SmoothQuant solves activation outlier challenges.

Quantization is the highest-ROI serving optimization — the difference between 2 GPUs and 1, or cloud-only and on-device. Combined with vLLM, TensorRT-LLM, or llama.cpp, it delivers dramatic cost and latency improvements.

Key Takeaways

• INT8 ~0.5% loss; INT4 ~1-3% loss — the practical sweet spot for LLMs
• Memory bandwidth, not compute, is the bottleneck — quantization reduces data movement
• Outlier handling separates success from failure — use AWQ, SmoothQuant, or LLM.int8()
• Tool choice by target: AWQ/GPTQ for GPU, GGUF for CPU/edge, bitsandbytes for experimentation
• Always evaluate on your specific task, not just perplexity