Your LLM app is slow? Buy a bigger GPU. Problem solved.
Just like a bigger oven fixes a restaurant with a bad kitchen manager.
Most of the speed in a modern LLM server doesn't come from the hardware. It comes from how the server decides which requests go on the GPU, when, and how much memory each one gets. Same model. Same GPU. The system around the model is what changes.
TL;DR. Generating tokens one at a time leaves a GPU mostly underused. Four ideas fix most of that: continuous batching (requests join and leave the batch at every step), paged KV cache memory (stop reserving memory you won't use), chunked prefill (long prompts stop freezing everyone else) and prefix caching (don't compute the same prompt twice). If you only call an API, the last one is the one you control, and it is mostly about prompt order.
First, why the GPU sits idle#
Every LLM request has two phases.
Prefill is when the model reads your whole prompt. All prompt tokens are processed together, in parallel. This is heavy work and it keeps the GPU busy.
Decode is when the model writes the answer, one token at a time. Each step produces just one new token per request. The Sarathi-Serve authors describe decode as having low compute utilization for exactly this reason, which is also why batching helps decode so much.
To avoid recomputing the whole conversation on every step, the model saves intermediate results for every token it has already seen. This is the KV cache (keys and values from the attention layers). It is big. The vLLM team measured up to 1.7 GB of KV cache for a single sequence on LLaMA-13B.
Now think of a naan wala with a tandoor that fits 20 naans. The tandoor is hot, the fuel is burning, and he bakes one naan at a time. Nineteen spots sit empty. That's decode with one request on a big GPU.
The obvious fix is to fill the tandoor. That is batching: run many requests through the same decode step. But how you fill it matters a lot.
Problem 1: the batch that waits for its slowest member#
The simple way is static batching. Collect a group of requests, run them together, and start the next group only when every request in the current group is finished.
The problem is that answers have different lengths. One user asks for a yes or no. Another asks for a 2,000-word essay. With static batching, the short answer is done in seconds but its slot stays empty until the essay finishes. New users wait outside.
Back to the tandoor. One customer wants a plain naan, which takes a minute. Another wants a stuffed kulcha, which takes longer. A naan wala doing static batching would wait for the kulcha before putting any new dough in. No naan wala works like that. The moment one naan comes out, new dough goes in.
That's continuous batching, also called iteration-level scheduling. The idea comes from Orca, a paper presented at OSDI 2022. Instead of scheduling a whole request from start to finish, the scheduler runs one decode step for the current batch, removes requests that finished, adds waiting requests if there is room, and repeats. A new request only waits for one iteration, not for a whole batch.
The authors evaluated it on GPT-3 175B and report a 36.9× throughput improvement over NVIDIA FasterTransformer at the same level of latency. That number is from their setup and their baseline, so don't expect 36.9× on your laptop. But it shows how much a dumb schedule wastes.

Every naan wala in Pakistan already knew this. We needed a 2022 paper lol.
Problem 2: memory, not compute, decides your batch size#
Continuous batching wants to keep adding requests. But every active request needs its own KV cache, and the KV cache grows with every token.
Older serving systems handled this badly. They didn't know how long an answer would be, so they reserved one big continuous chunk of GPU memory per request, sized for the maximum possible length. Most of it was never used. The vLLM team found that existing systems wasted 60% to 80% of KV cache memory because of this fragmentation and over-reservation.
It's like a restaurant that reserves a table for 10 for every customer, because maybe their friends will come. Most people come alone. The restaurant is "full" while half the chairs are empty.
PagedAttention fixes this the same way operating systems fixed memory decades ago: pages. The KV cache of each request is split into small fixed-size blocks, and the blocks don't need to sit next to each other in memory. Blocks are given out on demand, as the answer grows. According to the vLLM team, waste only happens in the last block of each sequence, which brings it under 4% in practice.
Less waste means more requests fit in memory at once. More requests means bigger batches. In the vLLM paper (SOSP 2023), the authors report 2× to 4× higher throughput than FasterTransformer and Orca at the same latency.
So the order of thinking is not "how fast is my GPU". It's "how many requests can my memory hold at once".
Problem 3: one long prompt freezes everyone#
Now a new problem. A user pastes a 50-page document. Its prefill is huge. If the scheduler runs that whole prefill in one go, every user who is in the middle of getting an answer stops receiving tokens until it's done.
The Sarathi-Serve paper calls these generation stalls, and shows that in vLLM (as it was then) they could last several seconds.
Tandoor version: someone orders 50 naans for a wedding. If the naan wala fills the whole tandoor with the wedding order, every regular customer in the queue just stands there.
The fix is chunked prefill. Split the long prompt into chunks and process one chunk per step, mixed in with the ongoing decodes. The wedding order goes in five naans at a time, next to the regular customers. Sarathi-Serve combines this with stall-free batching, where new requests join a running batch without pausing anyone already decoding. Its authors report serving capacity gains of up to 2.6× for Mistral-7B on one A100 and up to 6.9× for Falcon-180B on 8 A100s, compared with Orca and vLLM, under their latency targets.
The idea has since gone mainstream. In vLLM V1, chunked prefill is enabled by default whenever possible, and the scheduler puts pending decodes first, then fills the remaining token budget with prefill work.
There is a knob here, and it is a real trade-off. vLLM's docs explain that a smaller token budget per step gives better inter-token latency (fewer prefills slowing down decodes), and a larger one gives better time to first token. You pick based on what your users feel more.
Problem 4: computing the same prompt again and again#
Look at what most LLM apps actually send:
- A chatbot sends the same long system prompt with every message.
- A RAG app (retrieval-augmented generation, i.e. you fetch documents and paste them into the prompt) sends the same document with many different questions.
- A normal chat resends the whole conversation history on every new turn.
Without caching, the model recomputes the KV cache for all of that repeated text every single time.
Prefix caching keeps the KV cache of prompts it has already processed and reuses it when a new request starts with the same tokens. vLLM's design docs call it almost a free lunch, since it doesn't change model outputs, and note that public endpoints such as OpenAI and Anthropic use it too. vLLM matches by hashing each KV block together with all the tokens before it. SGLang does the same job with a radix tree (it calls this RadixAttention), which keeps the cache of finished requests around and finds the longest matching prefix for each new one.
Think of a restaurant's base gravy. The cook prepares the masala once in the morning, and every karahi that day starts from it. Nobody fries onions from scratch for every order.
But here is the catch, and it's the part you control. The match has to be at the start of the prompt, and it has to be exact. OpenAI's docs say it plainly: cache reuse requires the entire rendered prefix to match. Change one token early in the prompt and everything after it is a cache miss.
So if your system prompt starts with "Current time: 10:42:17", you just changed the masala recipe for every single order.

What this means for you, even if you never touch a GPU#
Most of us will never write a scheduler. Fine. But this changes how you should build.
If you call a hosted API, fix your prompt order. Put the stable things first: system instructions, tool definitions, shared documents, examples. Put the things that change last: timestamps, user names, the actual question. OpenAI recommends the same thing in its docs. I wrote about how to structure system prompts for production before, and this is one more reason that structure matters.
The money side is real. As of November 2026:
- OpenAI caches automatically. The minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later, and cached input tokens are discounted by up to 95%.
- Anthropic charges cache reads at 0.1× the base input price on most models (0.05× on Claude Opus 5.5 and Sonnet 5.5, 0.025× on Claude Fable 5.1 and Mythos 5.1). Writing to the cache costs 1.25× for a 5-minute cache or 2× for a 1-hour cache. You mark what to cache with
cache_control.
Prices change. Check the pricing pages before you build a budget on these numbers. If you want to put numbers on your own app, I have a guide on estimating AI agent API costs.
If you self-host, don't serve with a plain generate loop. Use an engine that already does all of the above, like vLLM or SGLang. Back in 2023, the vLLM team reported up to 24× higher throughput than plain Hugging Face Transformers in their benchmark. The exact number will differ for your model and traffic, but the gap comes from exactly the ideas in this post.
Where this argument breaks#
People say "just buy better GPUs". I partially agree.
- Batching improves throughput, not every user's speed. It lets the server finish more total work. A single user with no one else on the server gains nothing from a batch of one.
- Prefix caching only helps when prompts actually share a beginning and prefill is a big part of the work. vLLM's own docs say it brings no gain when most of the time goes into generating long answers, or when new queries don't share a prefix with old ones.
- Some problems really are hardware problems. If the model doesn't fit in memory, or you are already running full batches all day, scheduling can't create capacity that isn't there.
So the GPU still matters. It's just not the first thing to fix.
Fix the system before the hardware#
This is the same thing I keep saying about models. Don't jump to the most expensive frontier model when a well-chosen, cheaper one does the job. The system around the model matters more. Inference is the same story one layer down.
Before you upgrade the GPU, check three things. Is your serving engine doing continuous batching? Is your prompt order letting the cache work? And are you measuring the right number, throughput or latency, for what your users actually feel?
The naan wala doesn't buy a bigger tandoor when the queue gets long. He stops leaving spots empty.
References#
- Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., OSDI 2022
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., SOSP 2023
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention — vLLM blog, 2023
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., 2024
- Optimization and Tuning (chunked prefill) — vLLM documentation
- Automatic Prefix Caching design — vLLM documentation
- SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al.
- Prompt caching — OpenAI API documentation
- Pricing (prompt caching multipliers) — Claude API documentation



