Inference is the production phase of an AI model’s lifecycle — when a trained model processes new inputs and generates outputs. If training is learning to play chess by studying thousands of games, inference is sitting down at the board and playing. For large language models, inference means receiving a sequence of tokens (the prompt), running a forward pass through billions of parameters, and autoregressively generating output tokens one at a time, with each new token conditioned on all preceding tokens.
Inference is where AI meets the real world. Every ChatGPT response, every Midjourney image, every Copilot code suggestion is an inference call. As of 2026, global inference compute demand exceeds training compute by an estimated 10:1 ratio — and the gap is widening as AI moves from research curiosity to production infrastructure.
How LLM inference works
Understanding LLM inference requires grasping two distinct phases that happen during a single request:
Prefill (prompt processing). The model processes all input tokens in parallel through the transformer’s layers. This is compute-bound — the GPU performs matrix multiplications across every token simultaneously. For a 100K-token prompt on a 405B-parameter model, prefill involves trillions of floating-point operations. The output of this phase is a set of key-value (KV) pairs stored in GPU memory for every attention head at every layer, representing the model’s “understanding” of the prompt.
Decode (token generation). The model generates output tokens one at a time, each requiring a full forward pass through all layers. This phase is memory-bandwidth-bound rather than compute-bound — the bottleneck is reading model weights from GPU memory for each token, not the arithmetic itself. A single output token from a 70B-parameter model in FP16 requires reading approximately 140 GB of weights from memory, even though the actual computation is minimal.
This prefill-decode split explains why long prompts have high latency at the start (prefill) but then tokens stream out at a relatively steady rate (decode), and why optimizing inference requires fundamentally different strategies for each phase.
The inference optimization stack
Serving frontier models at scale requires a layered optimization approach. Each technique addresses a different bottleneck:
KV-cache management
During autoregressive decoding, attention computations from prior tokens are cached so they are not recomputed for every new token. For a 128-layer model with 128 attention heads and a 128K context window, the KV-cache alone can consume 50-100 GB of GPU memory per request. Techniques for managing this include:
- Paged attention (introduced by vLLM in 2023): Manages KV-cache memory in fixed-size pages, eliminating fragmentation and enabling memory sharing across requests. This single technique improved GPU utilization by 2-4x in multi-request serving.
- Multi-query attention (MQA) and grouped-query attention (GQA): Reduce KV-cache size by sharing key-value heads across multiple query heads. Llama 3 uses GQA with 8 KV heads shared across 64 query heads, cutting cache size by 8x.
- KV-cache compression: Quantizing cached values to INT8 or INT4, or selectively evicting low-importance entries, can halve memory requirements with minimal quality loss.
Quantization
Reducing the numerical precision of model weights decreases memory usage and increases throughput:
| Precision | Bits per parameter | Memory for 70B model | Typical quality impact |
|---|---|---|---|
| FP32 | 32 | 280 GB | Baseline (training) |
| FP16/BF16 | 16 | 140 GB | Negligible |
| INT8 (W8A8) | 8 | 70 GB | < 1% degradation |
| INT4 (GPTQ/AWQ) | 4 | 35 GB | 1-3% degradation |
| 2-bit (QuIP#) | 2 | 17.5 GB | 3-8% degradation |
INT4 quantization has become the standard for local deployment, enabling a 70B-parameter model to run on a single GPU with 48 GB of VRAM. The AWQ (Activation-aware Weight Quantization) and GPTQ methods are the most widely adopted, with AWQ generally preferred for its better preservation of outlier weights. FP8 inference has become the default on NVIDIA Hopper and Blackwell GPUs, offering near-lossless quality with 2x throughput gains over FP16.
Speculative decoding
A small “draft” model (e.g., 1-2B parameters) generates several candidate tokens quickly, and the larger “target” model verifies them in a single parallel forward pass. When the draft model’s predictions match the target model’s distribution — which happens 60-80% of the time for well-matched pairs — multiple tokens are accepted in one step. This can reduce decode latency by 2-3x without any quality loss, since rejected tokens are resampled from the correct distribution.
Google uses speculative decoding extensively in Gemini serving. Anthropic and OpenAI employ variants of this technique in production, though specific implementations are proprietary.
Continuous batching and request scheduling
Traditional static batching waits until a fixed batch of requests is assembled, then processes them together. Continuous batching (also called in-flight batching), pioneered by the Orca system in 2022, inserts new requests into a running batch as soon as slots free up. This keeps GPUs saturated rather than idle-waiting, improving throughput by 5-20x in high-traffic scenarios.
Modern inference engines like vLLM, TensorRT-LLM (NVIDIA), and SGLang implement sophisticated scheduling that combines continuous batching with paged attention and priority queues to maximize GPU utilization while meeting latency SLAs.
The inference hardware landscape
Inference hardware has become a strategic battlefield across the semiconductor industry:
| Hardware | Developer | Key advantage | Best for |
|---|---|---|---|
| H100 SXM | NVIDIA | Ecosystem maturity, FP8 Tensor Cores | General-purpose inference at scale |
| B200 / GB200 | NVIDIA | 2x H100 throughput, FP4 support | Next-gen large-model serving |
| TPU v5p | Tight integration with Gemini, cost efficiency | Google Cloud workloads | |
| Inferentia2 | Amazon | Low per-token cost, Neuron SDK | Cost-optimized AWS inference |
| Trainium2 | Amazon | Training + inference flexibility | Dual-purpose AWS deployment |
| LPU (Groq) | Groq | Deterministic latency, extreme speed | Latency-sensitive applications |
| Wafer-Scale Engine 3 | Cerebras | Single-chip model hosting, no parallelism overhead | Large models without partitioning |
| Apple Silicon (M4 Ultra) | Apple | Unified memory, on-device inference | Local/edge LLM inference |
Groq’s Language Processing Unit (LPU) deserves specific mention for its architectural novelty. By using synchronous, deterministic dataflow rather than traditional GPU scheduling, Groq achieves inference speeds exceeding 1,200 tokens per second on Llama 3 70B — roughly 10x faster than a single H100. The tradeoff is limited batch sizes and higher per-token costs at scale, making Groq best suited for latency-critical applications rather than throughput-maximized serving.
Inference cost economics
The cost of inference follows a power law: a small number of frontier-model requests consume a disproportionate share of compute budget. Understanding the cost structure is essential for production deployment:
Per-token pricing is the standard commercial model. As of mid-2026, representative prices per million tokens:
| Model | Input tokens | Output tokens | Output-to-input ratio |
|---|---|---|---|
| Claude Opus 4 | $15 | $75 | 5x |
| Claude Sonnet 4 | $3 | $15 | 5x |
| Claude Haiku 3.5 | $0.80 | $4 | 5x |
| GPT-4.1 | $2 | $8 | 4x |
| GPT-4.1 mini | $0.40 | $1.60 | 4x |
| Gemini 2.5 Pro | $1.25-2.50 | $10-15 | 6-8x |
| Llama 3.3 70B (self-hosted) | ~$0.20 | ~$0.20 | 1x |
Output tokens cost 4-5x more than input tokens because each output token requires a sequential forward pass, while input tokens are processed in parallel during prefill.
Self-hosted inference trades operational complexity for cost savings at scale. Running Llama 3.1 70B on a reserved H100 instance costs approximately $2-3 per hour. At high utilization (500+ requests per hour), self-hosted inference costs 5-10x less per token than API pricing. Below ~100 requests per hour, managed API services are almost always cheaper.
The model cascading pattern routes requests to the cheapest model capable of handling them. A typical implementation uses Haiku-class models for classification, extraction, and simple Q&A (80% of requests), Sonnet-class models for generation, analysis, and moderate reasoning (15%), and Opus-class models only for complex multi-step reasoning (5%). This can reduce blended inference costs by 70-85%.
Training vs. inference: the compute inversion
Historically, training consumed far more compute than inference. A single GPT-4-scale training run costs $50-100 million in compute. But this relationship has inverted:
A model trained once serves millions of users continuously. Meta’s Llama 3.1 405B cost an estimated $100 million to train, but serving it across the open-source ecosystem generates billions of inference calls monthly. By some estimates, within 12 months of deployment, cumulative inference compute for a widely used model exceeds its training compute by 100x or more.
This inversion explains why the AI industry’s center of gravity is shifting from training clusters to inference infrastructure. NVIDIA’s revenue increasingly comes from inference-oriented hardware. Startups like Groq, Cerebras, and SambaNova are inference-first companies. Cloud providers are racing to offer the lowest per-token costs for inference serving.
Edge and on-device inference
Not all inference happens in the cloud. A significant and growing share of inference runs on consumer devices:
Smartphones. Apple Intelligence runs models locally on iPhone 15 Pro and later using the Neural Engine. Google’s Gemini Nano runs on Pixel devices. Qualcomm’s Snapdragon 8 Gen 3 and Gen 4 chips include dedicated AI accelerators capable of running 7-13B parameter quantized models.
Laptops and desktops. Apple’s M-series chips with unified memory can run 30B+ parameter models at usable speeds. The M4 Ultra with 192 GB unified memory can host quantized versions of models up to 200B+ parameters. NVIDIA’s RTX 4090 and 5090 handle 70B quantized models. Tools like llama.cpp, Ollama, and LM Studio make local inference accessible to non-technical users.
Embedded and IoT. TinyML inference runs sub-100M parameter models on microcontrollers for tasks like keyword detection, gesture recognition, and anomaly detection, using frameworks like TensorFlow Lite Micro and ExecuTorch.
The primary advantages of edge inference are latency (no network round trip), privacy (data never leaves the device), cost (no per-token charges), and offline availability. The primary constraint is model size — the most capable models still require datacenter-class hardware.
Inference serving architectures
Production inference systems use several architectural patterns:
Single-model serving. A model runs on one or more GPUs behind a load balancer. Simple, but limited in throughput. Suitable for low-traffic applications.
Tensor parallelism. A single model is split across multiple GPUs within a node, with each GPU holding a slice of every layer. Required for models too large to fit on a single GPU. Llama 3.1 405B in FP16 requires at least 8x H100 80GB GPUs.
Pipeline parallelism. Different layers of the model run on different GPUs or nodes. Increases throughput by overlapping prefill and decode across pipeline stages, but adds latency for individual requests.
Disaggregated serving. Prefill and decode phases run on separate hardware optimized for each. Prefill runs on compute-dense GPUs; decode runs on memory-bandwidth-optimized hardware. This architecture, deployed in production by several hyperscalers, can improve total cost-efficiency by 30-50%.
Multi-model routing. A lightweight router model or rule-based system directs each request to the optimal backend model based on complexity, cost, and latency requirements. This enables the model cascading pattern described above.
Frequently asked questions
What is the difference between training and inference in AI? Training is the process of teaching a model by adjusting its weights on large datasets — it happens once (or periodically) and is extremely expensive, often costing millions of dollars for frontier models. Inference is using the trained model to generate outputs from new inputs — it happens continuously in production and scales with usage. Training is compute-bound and runs for weeks; inference is memory-bandwidth-bound and runs in milliseconds to seconds.
Why is inference so expensive for large language models? LLM inference is expensive because each output token requires a sequential forward pass through billions of parameters, and the weights must be read from GPU memory for every token. A 70B model in FP16 requires reading 140 GB of data from memory per token. At scale, the GPU-hours add up: a service handling 1 million requests per day on a frontier model may need 100+ GPUs running continuously. Optimization techniques like quantization, speculative decoding, and model cascading can reduce costs by 50-90%.
Can I run AI inference on my own hardware? Yes. Quantized open-source models can run on consumer hardware. A 7-13B parameter model in INT4 requires 4-8 GB of VRAM and runs well on recent gaming GPUs or Apple Silicon Macs. A 70B model in INT4 needs roughly 40 GB of VRAM (e.g., an RTX 4090 or M-series Mac with 48+ GB unified memory). Tools like Ollama, llama.cpp, and LM Studio make setup straightforward. Quality is lower than frontier API models but sufficient for many applications.
What is speculative decoding? Speculative decoding is an inference acceleration technique where a small, fast “draft” model generates several candidate tokens, and the large target model verifies them all in a single forward pass. When the draft model’s predictions are correct — which happens the majority of the time — multiple tokens are accepted at once, reducing effective latency by 2-3x with no quality degradation. It is widely used in production serving by major AI providers.
How fast will inference get? Inference speed is improving on multiple fronts simultaneously: hardware (each GPU generation delivers 2-3x more inference throughput), software (techniques like speculative decoding and disaggregated serving), and architecture (mixture-of-experts models that activate fewer parameters per token). Over the past two years, the cost per million tokens has dropped roughly 10x for equivalent quality levels, and this trend is expected to continue. By 2027, real-time voice conversation with frontier-quality models is expected to become computationally trivial.