Skip to main content

​
Contextual Retrieval

A RAG preprocessing technique that prepends chunk-specific context to each chunk before embedding and BM25 indexing — preserving document provenance that traditional chunking strips away.

​
The problem it solves

Standard RAG chunks split documents at token boundaries with no awareness of where they came from. Chunks become decontextualized fragments. Retrieval fails when the query’s intent maps to document-level context that no longer exists in the chunk.

​
Mechanism

For each chunk, an LLM generates a short (50-100 token) context string that situates the chunk within its source document. This context is prepended to the chunk text before both:
  • Vectorization (Contextual Embeddings)
  • BM25 index construction (Contextual BM25)
The chunk itself is unchanged. Only the indexed representation gains context.

​
Why both BM25 and embeddings

Embeddings capture semantic similarity; BM25 captures exact lexical matches. Neither alone is sufficient:
  • Embeddings miss exact identifiers (“error code TS-999”, specific names)
  • BM25 misses paraphrase and semantic proximity
Applied together with context, their weaknesses don’t overlap.

​
Cost model

LLM context generation is the expensive step. With prompt caching (full document cached, only chunk varies per call): ~$1.02/million document tokens using Claude 3 Haiku.

​
Performance

−49% retrieval failure vs baseline (BM25 + embeddings without context). Adding reranking: −67%.

​
Code search: a harder variant of the same problem

Greptile (2024) measured why semantic search on codebases underperforms text search: code and natural language queries are semantically distant. Noise compounds the problem: embedding a full file containing the correct function returns similarity 0.739 — barely better than embedding random code (0.718) and much worse than the function alone (0.768). Solutions mirror contextual retrieval:
  1. Translate code to natural language summary before embedding (analogous to prepending context)
  2. Chunk at function level, not file level (smaller, focused units = less noise dilution)
Why the wiki avoids this problem: wiki pages are natural-language descriptions of concepts. No translation step needed — each page is already the “NL description” that achieves 0.815-class similarity. Source: raw/Codebases are uniquely hard to search semantically.md
  • BM25 — the lexical retrieval half of the pipeline
  • Reranking — post-retrieval filtering that stacks with contextual retrieval gains
  • Compounding Knowledge Base — alternative pattern where context is pre-compiled into wiki pages rather than prepended at index time
  • qmd — local search engine using the BM25 + vector hybrid this technique validates