Quantization reduces the numerical precision of a model’s weights and activations — converting 16-bit floating-point numbers (FP16) to 8-bit integers (INT8), 4-bit integers (INT4), or even lower — to shrink the model’s memory footprint and accelerate inference. A 70-billion-parameter model stored in FP16 requires approximately 140 GB of GPU memory; quantized to 4-bit, it fits in roughly 35 GB, making it runnable on a single high-end consumer GPU instead of requiring a multi-GPU server. Quantization is the single most important technique enabling local AI — it is the reason models that were trained on clusters of thousands of GPUs can run on a laptop.
The fundamental insight behind quantization is that neural network weights are massively over-precise. Most weights cluster near zero and vary by small amounts. Storing each one as a 16-bit or 32-bit floating-point number preserves far more precision than the model actually uses. By mapping these weights to a smaller set of representable values, quantization trades imperceptible precision for transformative efficiency gains.
How quantization works
To understand quantization, start with how numbers are stored. A 16-bit floating-point number (FP16) can represent approximately 65,000 distinct values. An 8-bit integer can represent 256 values. A 4-bit integer can represent only 16 values. Quantization maps the continuous range of FP16 weight values onto these smaller discrete sets.
Linear quantization is the simplest approach. It takes the minimum and maximum weight values in a layer, divides the range into equal intervals, and maps each weight to the nearest interval. For 8-bit quantization, this means dividing the range into 256 bins. Each weight is stored as an 8-bit index plus two FP16 scale factors (min and max), reducing storage by roughly 50% while enabling integer arithmetic during inference, which is 2-4x faster than floating-point on most hardware.
Group quantization improves on linear quantization by applying the mapping to small groups of weights (typically 32-128) rather than entire layers. Each group gets its own scale factor, allowing better representation of weight distributions that vary across the layer. GPTQ and AWQ both use variants of group quantization. This adds modest storage overhead (one scale factor per group) but significantly improves quality.
Non-uniform quantization uses lookup tables or learned codebooks to represent weights, allowing the quantization bins to be unevenly spaced. Since most weights cluster near zero, placing more bins near zero and fewer at the extremes preserves more information. Techniques like QuIP# and AQLM use sophisticated non-uniform approaches to push quality at very low bit widths (2-3 bits).
The quantization methods landscape
The open-source community has developed numerous quantization methods, each with distinct tradeoffs:
| Method | Bit widths | Approach | Quality retention | Speed | Best for |
|---|---|---|---|---|---|
| GPTQ | 2-8 bit | Post-training, group quantization | Very good at 4-bit | Fast on GPU | GPU inference with maximum quality |
| AWQ | 4 bit | Activation-aware weight quantization | Excellent at 4-bit | Fast on GPU | Production GPU deployment |
| GGUF/llama.cpp | 2-8 bit | CPU-optimized mixed quantization | Good across range | Good on CPU+GPU | Local inference, consumer hardware |
| ExLlamaV2 | 2-8 bit | GPU-optimized with mixed precision | Excellent at lower bits | Fastest on GPU | Maximum throughput on GPU |
| SmoothQuant | 8 bit | Weight-activation co-quantization | Excellent | Fast | INT8 deployment on standard hardware |
| bitsandbytes | 4-8 bit | Dynamic quantization during loading | Good | Moderate | Easy integration with HuggingFace |
| HQQ | 2-4 bit | Half-quadratic quantization | Good at very low bits | Fast | Ultra-low-bit research |
Post-training quantization vs. quantization-aware training
The two fundamental approaches to quantization differ in when the precision reduction happens:
Post-training quantization (PTQ) takes a fully trained FP16 model and converts it to lower precision without additional training. The process typically takes minutes to hours on a single GPU, making it accessible to anyone with the model weights. PTQ is the standard approach in the open-source community — when someone on HuggingFace uploads a “GPTQ” or “AWQ” version of a model, they have applied post-training quantization.
The quality of PTQ depends heavily on the method used. Naive round-to-nearest quantization at 4-bit causes significant degradation (5-15% quality loss). Sophisticated methods like GPTQ and AWQ use calibration datasets (typically 128-1024 samples of representative text) to minimize quantization error, achieving 4-bit with only 1-3% quality loss on most benchmarks.
Quantization-aware training (QAT) incorporates simulated quantization during the training or fine-tuning process. The model learns to produce weights that quantize well, effectively adapting its weight distributions to the target precision. QAT produces higher-quality quantized models than PTQ — typically 0.5-2% better at the same bit width — but requires retraining the model, which costs thousands of GPU-hours.
QAT is used primarily by model developers with training resources. Google’s Gemma models and Apple’s OpenELM models include QAT as part of their training pipeline. For most practitioners who are quantizing existing models, PTQ is the practical choice.
Memory and speed impact by precision level
The concrete numbers matter for deployment planning:
| Precision | Memory per 70B model | Relative quality (vs FP16) | Inference speed multiplier | Typical hardware |
|---|---|---|---|---|
| FP32 | ~280 GB | Baseline (identical) | 0.5x (slower) | Training only |
| FP16/BF16 | ~140 GB | 100% (reference) | 1x | Multi-GPU servers (2x A100-80GB) |
| FP8 | ~70 GB | 99-99.5% | 1.5-2x | Single A100-80GB or H100 |
| INT8 | ~70 GB | 98-99.5% | 2-3x | Single A100/H100 |
| INT4 (GPTQ/AWQ) | ~35 GB | 95-98% | 2-4x | Single A100 or 2x RTX 4090 |
| INT4 (GGUF) | ~35 GB | 94-97% | 1.5-3x (CPU+GPU) | Gaming PC, Mac with 64GB RAM |
| INT3 | ~26 GB | 88-94% | 2-4x | Consumer GPU (RTX 4090) |
| INT2 | ~18 GB | 80-90% | 3-5x | Research/experimental only |
These numbers explain why 4-bit quantization has become the sweet spot for local inference. A 70B model at 4-bit fits on hardware that costs $2,000-4,000 (a single RTX 4090 with 24GB VRAM using partial CPU offloading, or an M-series Mac with 64GB unified memory), compared to $30,000+ for the multi-GPU setup needed for FP16.
Quantization in the inference stack
Quantization does not exist in isolation — it integrates with inference frameworks that optimize the full pipeline:
NVIDIA TensorRT-LLM supports FP8 and INT8 quantization with hardware-accelerated kernels on H100 and later GPUs. NVIDIA’s FP8 format, introduced with the Hopper architecture, is specifically designed for AI inference — it provides a good balance of range and precision that minimizes quantization error. TensorRT-LLM with FP8 on H100 achieves roughly 2x the throughput of FP16, with negligible quality loss.
vLLM, the most popular open-source LLM inference server, supports GPTQ, AWQ, and bitsandbytes quantization formats. It combines quantization with PagedAttention (an efficient memory management technique for KV cache) to maximize throughput. A vLLM deployment with AWQ 4-bit quantization can serve 3-5x more concurrent users than the same hardware running FP16.
llama.cpp is the dominant framework for local inference, supporting GGUF-format quantization that mixes precision across layers. Its key innovation is efficient CPU+GPU split inference — the model can be partially loaded on a GPU with the remainder on CPU RAM, enabling inference on hardware that cannot fit the entire model in GPU memory. This is how most consumer deployments of 70B models work.
Apple MLX brings quantized inference to Apple Silicon, leveraging the unified memory architecture of M-series chips. A 4-bit quantized 70B model can run on an M4 Max MacBook Pro with 128GB unified memory, achieving 15-25 tokens per second — usable for interactive applications. MLX supports 4-bit and 8-bit quantization with quality comparable to GGUF.
Real-world quantization deployments
Local AI on consumer hardware. The local AI movement runs almost entirely on quantized models. Applications like LM Studio, Ollama, and Jan provide one-click deployment of quantized models on consumer hardware. A 4-bit quantized version of Llama 3.1 70B runs at 10-20 tokens/second on a gaming PC with an RTX 4090 — slow by cloud standards, but perfectly usable for coding assistance, writing, and analysis. The community on r/LocalLLaMA has tested thousands of quantization configurations, establishing practical guidelines for which models and methods work best on specific hardware.
Cloud inference optimization. Cloud providers use quantization to reduce serving costs. Google serves some Gemini API traffic using INT8 quantized models on TPUs. Together AI and Fireworks AI offer quantized model endpoints at lower prices — Together AI’s Llama 3.1 70B endpoint uses INT4 quantization to offer pricing approximately 60% lower than full-precision serving while maintaining benchmark parity.
Edge and mobile deployment. Qualcomm’s AI Engine on Snapdragon 8 Gen 3 processors supports INT4 and INT8 inference for models up to 10B parameters. Google’s Gemini Nano, deployed on Pixel phones, uses quantization to run a capable model in under 4GB of memory. Samsung’s Galaxy AI features use quantized on-device models for translation, summarization, and image editing.
Autonomous vehicles and robotics. NVIDIA’s DRIVE platform uses INT8 quantized models for real-time perception and planning. The latency requirements (inference in under 10ms) make quantization mandatory — FP16 models simply cannot meet timing constraints on embedded hardware. Tesla’s Full Self-Driving computer uses aggressive quantization for its neural networks.
The quality-precision frontier
The relationship between precision and quality is not linear. Research has established several important patterns:
The 4-bit threshold. Across dozens of model architectures and sizes, 4-bit quantization consistently retains 95-98% of full-precision quality. Below 4-bit, degradation accelerates non-linearly. This threshold exists because 4 bits (16 values) provides just enough resolution to represent the weight distributions typical of transformer models. At 3 bits (8 values), important distinctions between weights are lost.
Size-dependent sensitivity. Larger models are more robust to quantization than smaller ones. A 70B model quantized to 4-bit typically retains more capability than a 7B model at the same precision, because the larger model has more redundancy in its weights. This is why 4-bit quantized 70B models often outperform full-precision 13B models — the extra parameters compensate for precision loss.
Task-dependent degradation. Quantization affects different capabilities unevenly. Factual recall and general conversation are highly robust (under 1% degradation at 4-bit). Mathematical reasoning and coding are moderately affected (2-5% degradation). Instruction following for complex, multi-constraint prompts is most sensitive (3-8% degradation). This means the acceptable quantization level depends on the deployment use case.
Activation outliers. Some transformer layers produce activation values far outside the normal range (“outliers”), and these values are critical for model quality. Standard quantization crushes these outliers, causing disproportionate quality loss. Methods like SmoothQuant and LLM.int8() specifically handle outliers by treating them separately, enabling clean INT8 quantization with minimal quality impact.
Quantization vs. distillation: when to use which
These two compression techniques serve different purposes and are often confused:
Use quantization when you have a specific model and want to deploy it more efficiently without changing its architecture. Quantization preserves the exact model structure and most of its capabilities while reducing memory and compute requirements. The process takes minutes to hours and requires no training data.
Use distillation when you need a fundamentally smaller model — fewer layers, fewer parameters, different architecture. Distillation produces a new model that is architecturally different from the teacher. The process takes days to weeks and requires large training datasets but can achieve much higher compression ratios.
Use both when deploying on severely constrained hardware. A 405B model distilled to 8B and then quantized to 4-bit produces a model that fits in ~4 GB of memory — runnable on a phone — while retaining useful capabilities for targeted tasks.
Emerging quantization research
The field continues advancing rapidly:
1-bit models (BitNet). Microsoft Research’s BitNet architecture trains models with ternary weights (-1, 0, 1) from scratch, rather than quantizing post-training. BitNet b1.58 matches FP16 transformer performance at 7B parameters while requiring only 1.58 bits per weight. If this approach scales, it could enable 70B-equivalent models to run on phones.
Mixed-precision quantization. Rather than quantizing all layers to the same precision, mixed-precision approaches assign higher precision to sensitive layers and lower precision to robust ones. The GGUF format supports this natively with its “Q4_K_M” and similar types, where different layer groups get 4-bit, 5-bit, or 6-bit precision based on sensitivity analysis.
KV cache quantization. During inference, the key-value cache (storing attention computations for previous tokens) can grow extremely large for long contexts. Quantizing the KV cache to 4-bit or 8-bit reduces memory usage proportionally, enabling longer context lengths on the same hardware. This is distinct from weight quantization and is increasingly important as context windows expand.
Frequently asked questions
Does quantization change the model itself? Quantization does not retrain or fundamentally change the model — it converts the numerical representation of existing weights to lower precision. The model’s architecture, layer structure, and learned patterns remain the same. The only change is the precision at which weights are stored and computed, which introduces small rounding errors that slightly affect outputs.
What is the best quantization method for running models locally? For GPU inference, AWQ and GPTQ at 4-bit provide the best quality-speed balance. For CPU or mixed CPU+GPU inference (common on consumer hardware), GGUF format with llama.cpp is the standard. For Apple Silicon Macs, MLX with 4-bit quantization is optimized for the unified memory architecture. The choice depends primarily on your hardware.
How much quality do you lose with 4-bit quantization? On standard benchmarks (MMLU, HumanEval, GSM8K), 4-bit quantization using modern methods like GPTQ or AWQ typically shows 1-3% degradation compared to FP16. In practice, most users cannot distinguish 4-bit outputs from full-precision outputs in conversation, coding assistance, and analysis tasks. Mathematical reasoning and complex instruction following show the most noticeable impact.
Can you quantize any model? Any transformer-based model can be quantized, but results vary. Models specifically trained or fine-tuned with quantization in mind (through QAT or architecture choices) quantize better. Models with highly variable weight distributions or many activation outliers may degrade more at low precision. In practice, all major open-weight models (Llama, Mistral, Qwen, Gemma, Phi) quantize well to 4-bit with modern methods.
Why not just train smaller models instead of quantizing large ones? A quantized large model and a natively small model with the same memory footprint do not have the same capabilities. The large model was trained with more parameters and captured more knowledge — quantization compresses the representation but retains much of that knowledge. A 4-bit 70B model typically outperforms a full-precision 7-13B model on most benchmarks, because the 70B model’s greater parametric knowledge survives the precision reduction.