Skip to main content

​
BM25 (Best Matching 25)

Lexical ranking function for information retrieval. Finds exact word/phrase matches. Complements semantic embeddings in hybrid search systems.

​
How it works

Extends TF-IDF (Term Frequency–Inverse Document Frequency):
  • TF-IDF: measures how important a word is to a document relative to a corpus — high frequency in doc, low frequency across corpus = high score
  • BM25 refinements: applies a saturation function to term frequency (prevents high-frequency common words from dominating) and normalizes for document length
Result: a relevance score per document for a given query, based purely on lexical overlap.

​
Where embeddings fail, BM25 succeeds

Semantic embeddings generalize — useful for meaning-based retrieval but lose specificity. BM25 finds exact strings:
  • Error codes: "TS-999", "HTTP 429"
  • Proper names, product names, version numbers
  • Technical identifiers without semantic neighbors

​
Where BM25 fails, embeddings succeed

BM25 requires lexical overlap. Paraphrase, synonyms, cross-lingual queries, and conceptual similarity without shared vocabulary all evade BM25. Standard pattern: retrieve top-K via BM25, retrieve top-K via embeddings, combine with rank fusion (e.g., Reciprocal Rank Fusion), deduplicate. Best of both retrieval modes. Used in: qmd, Contextual Retrieval, most production RAG pipelines.
  • Contextual Retrieval — adds document context to BM25 index entries to reduce context-stripping failures
  • qmd — local search engine using BM25 + vector hybrid