Context Assembler in Machine Learning

Let's dive in! The retrieval stage of a RAG pipeline produces a ranked list of candidate passages. Sounds great. But you absolutely cannot just naively concatenate them and toss them at the LLM. Here's why -- and I'll walk through each reason because they all matter.

Context windows are finite and expensive

As of 2025, frontier models support windows ranging from 8K to 200K tokens. But cost scales linearly with input token count. A 128K-token prompt can cost 10-50x more than a 4K-token prompt for the exact same query. With GPT-4o at $2.50 per million input tokens, sending 15,000 tokens when 4,000 would suffice costs you roughly$0.0275 extra per 1,000 queries ($2.30 or about INR 192). That adds up fast at scale. Gao et al. (2023) categorize this as the augmentation bottleneck in their taxonomy of RAG paradigms.

Passage ordering materially affects generation quality

This one surprises people. Liu et al. (2024) showed that models like GPT-3.5-Turbo and Claude 2 exhibit a U-shaped performance curve: they attend most effectively to information at the beginning or end of the context, with accuracy dropping by up to 20 percentage points for evidence buried in the middle. Without deliberate ordering, you're leaving answer quality to chance.

Retrieved passages frequently contain overlapping content

Especially when chunks are created with overlap windows during ingestion (which is standard practice), the same paragraph can appear in 3-4 consecutive chunks. Sending all that redundancy wastes tokens and can confuse the generator when the same fact appears in slightly different phrasings.

Production systems require source attribution

Users and audit systems need to trace each claim back to a specific source document. The context assembler must inject passage identifiers, source URLs, or confidence scores so the generator can produce inline citations. Without this, you get an answer nobody can verify.

Format translation between retriever and generator

Different LLMs expect different prompt structures -- chat message arrays for OpenAI, XML-tagged sections for Anthropic, markdown-delimited blocks for others. The assembler handles this translation so your retrieval pipeline stays decoupled from your generator choice.

If you take away one thing: the context assembler exists because the gap between "I have good passages" and "the LLM produces a good answer" is wider than most teams realize.

I like to think of the context assembler as a newspaper editor who receives a stack of wire reports (retrieved passages) and must lay out a single page (the prompt) for a reader (the LLM) with limited attention.

The editor must decide which stories to run, in what order, trimming redundant paragraphs, making sure critical facts appear where the reader's eye naturally falls -- the headline and the closing paragraph -- and attaching bylines for attribution.

The hard budget constraint

Here's the core insight: the context assembler operates under a hard budget constraint. The total token count across the system prompt, assembled context, and user query must not exceed the model's context window, and should ideally stay well below it to leave room for the generated response.

This transforms what looks like a simple formatting task into an optimization problem: maximize the information density of the assembled context within a fixed token envelope, while respecting ordering constraints that account for the model's positional attention biases.

What makes it valuable

The assembler doesn't generate new information -- it curates. That was pretty simple, wasn't it? But the simplicity is deceptive. Its value lies entirely in:

1. What it includes -- the highest-signal passages
2. What it excludes -- noise, redundancy, low-value content
3. The structure it imposes -- ordering for attention, metadata for citations

A well-designed context assembler ensures the generator sees the most relevant evidence, in the most favorable positions, with minimal redundancy and clear source provenance -- all within a budget that balances quality against inference cost.

Think: curate ruthlessly, order deliberately, attribute clearly.

Let's build up the math. Don't worry -- the intuition comes first, the formulas just make it precise.

Token budget equation

The assembler's first job is figuring out how many tokens it has to work with. The system prompt eats some tokens, the user query eats some, and we need to reserve space for the model's answer. Whatever's left is our context budget.

$T_{context} = T_{max} - T_{sys} - T_{query} - T_{reserved}$

where $T_{reserved}$ is a margin reserved for the model's generated output (typically 256-2048 tokens depending on the task).

The selection and ordering problem

Given a ranked list of $n$ retrieved passages $P = [p_1, p_2, \ldots, p_n]$ where each $p_i$ has token count $t_i$ and relevance score $s_i$, the context assembler solves a constrained selection and ordering problem:

$\text{Maximize:} \quad \sum_{p_i \in S} s_i \cdot w(\text{position}(i))$

$\text{Subject to:} \quad \sum_{p_i \in S} t_i \leq T_{context}$

where $w(\text{position}(i))$ is a positional weight function that accounts for the model's attention bias -- assigning higher weight to the first and last positions in the context block to counteract the lost-in-the-middle effect.

How it works in practice

In practice, greedy algorithms suffice. Passages are selected in rank order until the budget is exhausted, then reordered so the highest-scoring passages occupy the primacy and recency positions. More sophisticated approaches apply context compression (Jiang et al., 2023) to reduce $t_i$ for each passage before selection, effectively fitting more evidence into the same budget.

The deduplication constraint

The deduplication constraint adds a pairwise similarity condition: for any two selected passages $p_i$ and $p_j$, their content overlap (measured by n-gram overlap, Jaccard similarity, or embedding cosine similarity) must fall below a threshold $\theta$, or the lower-ranked passage is removed.

The formal problem looks like a constrained knapsack with ordering -- and that's essentially what it is.

Let's recap what we've covered about the context assembler.

• The context assembler is the post-retrieval module that constructs the final LLM prompt from retrieved passages, managing token budgets, passage ordering, deduplication, and source attribution. It's your last quality gate before an expensive inference call.

• Token budget allocation is the foundational constraint: $T_{context} = T_{max} - T_{sys} - T_{query} - T_{reserved}$, computed using the model's exact tokenizer (not len(text.split()) -- please).

• Passage ordering counteracts the lost-in-the-middle effect (Liu et al., 2024) by placing the highest-value passages at the beginning and end of the context block, where LLMs attend most effectively. This is a 10-20% accuracy boost that costs you nothing.

• Deduplication prevents overlapping chunks from wasting token budget on redundant content -- critical when chunks are created with overlap windows, which is standard practice.

• Context compression techniques (LongLLMLingua, RECOMP) can fit 2-6x more information into the same token budget, trading modest compute for significant cost savings.

• Source attribution is a context assembly responsibility: passage identifiers must be injected into the context so the LLM can produce verifiable citations. No metadata in, no citations out.

• Cost scales linearly with context length -- a well-tuned context assembler that selects only high-value passages can reduce inference cost by 50-75% compared to naive concatenation. At scale, that's the difference between a viable product and an unsustainable one.

The context assembler is the quality and cost control surface of a RAG pipeline. It determines what evidence the LLM sees, in what order, and at what cost -- making it a deceptively critical component that warrants careful engineering and evaluation.