How to Budget Retrieved Context in a RAG Pipeline

A useful RAG token budget starts with the complete request, not just the retrieved passages. Reserve capacity for system instructions, the user query, conversation history, retrieval metadata and formatting, and the model’s output allowance. The remaining space is the maximum available for retrieved context—and it should be treated as a ceiling, not a target that every request must fill.

There is no universal chunk size or top-k value that works for every retrieval-augmented generation (RAG) workload. Choose chunking and retrieval settings based on your model’s limits, document structure, query patterns, relevance, answer quality, and latency or cost constraints. Then measure the assembled request with the tokenizer or request-counting API for the model you actually call.

Calculate the budget before retrieval#

Start with the model’s documented context capacity and output constraints. The exact relationship varies by model and API: some limits share a total context budget, while others describe input and output separately. Confirm the model’s current documentation rather than assuming every provider exposes limits in the same way.

Plain text

retrieval_budget = context_limit
                 - instructions_and_query
                 - conversation_history
                 - formatting_and_metadata
                 - reserved_output
                 - safety_margin

This is a planning model, not a substitute for the API’s actual validation rules. If your provider defines separate input and output limits, enforce both. If tools, images, or structured message boundaries add tokens, count them through the provider’s supported request-level method where possible.

A conceptual RAG context budget split into instruction and query overhead, retrieved passages, output reserve, and safety margin. — Conceptual allocation only; actual limits depend on the model and API.

Measure the request you will actually send#

A retrieved document chunk is not always the same string that reaches the model. The application may add a source title, URL, document ID, rank, access-control metadata, separators, instructions, or a citation wrapper. Those fields can be useful, but they also consume input capacity.

Count the final serialized context after the application has assembled these parts. Counting each chunk in isolation is useful for filtering candidates, but it cannot account for repeated wrappers, query text, chat history, or message structure.

Use the Letinix Token Calculator for a quick text-only inspection. It provides exact counts for the o200k_base and cl100k_base encodings and a character-based estimate for other selected models. For a production request, use the target provider’s request-counting API when it supports the inputs you send, and compare the count with actual usage reported by completed requests.

Treat chunk size and retrieval count as separate decisions#

Chunk size determines how much source material is grouped into each retrieval unit. Retrieval count determines how many candidates are selected. They interact, but changing both at once makes it harder to identify why quality or cost changed.

Small chunks can make retrieval more specific but may separate a claim from the context needed to interpret it. Large chunks can preserve surrounding context but may include irrelevant passages and consume more of the prompt budget. Overlap may preserve information near boundaries, but it can also repeat text across retrieved chunks.

The right values depend on the corpus and the question. A policy manual, API reference, source-code repository, and conversational knowledge base have different structural boundaries and retrieval needs. Avoid treating a provider’s example configuration as a universal recommendation.

A controlled evaluation should vary one dimension at a time:

Establish a baseline with the current chunking and retrieval settings.
Change chunk size or overlap while holding the retrieval count and prompt constant.
Measure retrieval relevance and answer quality on a representative query set.
Change the retrieval count and evaluate again.
Compare token use, latency, failure modes, and answer quality—not token reduction alone.

Google’s Gemini File Search documentation, for example, exposes chunking configuration with maximum tokens per chunk and maximum overlap tokens. Those parameters are configuration options, not evidence that a particular setting is best for another corpus or model.

Select chunks under a token ceiling#

A fixed top-k retrieval step can return a set whose combined serialized size exceeds the available budget. A budget-aware assembly step should count candidates and include them only while the request remains within its limit.

A simple greedy strategy can work as a first implementation:

rag-budget.tsTypeScript

// Pseudocode: countWithTargetModel must use the target model's tokenizer.
const selected = [];
let used = countWithTargetModel(renderPrompt({ query, chunks: [] }));

for (const chunk of rankedChunks) {
  const candidate = renderPrompt({ query, chunks: [...selected, chunk] });
  const candidateTokens = countWithTargetModel(candidate);

  if (candidateTokens + reservedOutput + safetyMargin <= contextLimit) {
    selected.push(chunk);
    used = candidateTokens;
  }
}

This example illustrates a budget check; it is not a tested production implementation. In a real system, use the provider’s actual input/output constraint semantics, avoid repeatedly re-tokenizing the entire prompt when that becomes expensive, and preserve any required citation or permission metadata. A greedy fit check also does not guarantee that the selected chunks are the most useful combination: relevance, diversity, source coverage, recency, and redundancy may need to influence selection.

If chunks are reordered or truncated after retrieval, retain source boundaries and enough provenance for citations. Never trim text blindly in a way that leaves a number, exception, code block, or claim without the qualification needed to interpret it.

Evaluate the budget against answer quality#

The goal is not to maximize the number of tokens sent. It is to include enough relevant evidence to answer the query while avoiding unnecessary or redundant context.

Retrieval quality: whether the needed source passages appear among candidates and selected chunks.
Answer quality and grounding: whether the answer uses the retrieved evidence correctly and preserves qualifications.
Context utilization: actual input tokens relative to the available input budget.
Operational behavior: latency, request failures, truncation, and cost under realistic traffic.

A smaller prompt is not necessarily a better RAG system if it drops decisive evidence. Likewise, a larger prompt is not automatically better if it adds duplicates or distractors. Token usage is one constraint in the evaluation, not the quality metric by itself.

For cost-focused planning after you have measured representative input and output usage, use the LLM API Cost Calculator. It estimates spend from request volume and token counts; it does not replace a model-specific count or an evaluation of retrieval quality.

A practical implementation checklist#

Confirm the target model’s context and output limits from its current documentation.
Measure system instructions, query, history, metadata, and serialized retrieved passages—not only raw chunks.
Keep output allowance and a safety margin separate from the retrieval budget.
Tune chunk size, overlap, and retrieval count independently before testing combinations.
Enforce a budget after retrieval, using a target-model-compatible counter or request-counting API.
Evaluate relevance, grounding, quality, latency, and cost together on representative queries.
Re-run the evaluation after changing the model, prompt template, corpus, chunker, or metadata format.