BM25 walkthrough

BM25 is a probabilistic ranking function that is widely used in Information Retrieval (IR) and search engines (ElasticSearch or Hybrid Search in RAG).

It’s an evolution of traditional TF-IDF (Term Frequency - Inverse Docuemnt Frequency) designed to fix TF-IDF main weakness: unbounded term frequency scaling and bias towards longer documents.

Core Formula

For a query Q consisting of terms q1, q2, q(n), and a candidate document D, the BM25 score is computed as follows:

Where:

• $f(q_i, D)$ is the frequency of query term $q_i$ in document $D$.
• $\vert{}D\vert{}$ is the length of document $D$ (total word count).
• $\text{avgdl}$ is the average document length across the entire corpus.
• $k_1$ and $b$ are free hyperparameter constants.

IDF Component

Where:

• $N$ is the total number of documents in the corpus.
• $n(q_i)$ is the number of documents containing the term $q_i$.

Parameter	Typical Range	What it Controls	When to Increase
k1​	1.2−2.0	Term frequency saturation speed.	When multiple occurrences of a term in a long text genuinely add more relevance signal.
b	0.75	Sensitivity to document length.	When your corpus has highly variable document lengths and long texts tend to dilute relevant content.

Recap:

BM25 depends on 3 pillars:

• Rarity: The rarer the word, the higher the score. A rare word acts as a heavy weight multiplier, while a common word is down-weighted to near zero. BM25 handles it by using the IDF (inverse document frequency) method.

• Term Saturation: Handles instances where seeing a query word once indicates that the document is relevant. Seeing it 3-4 times confirms that it’s heavily focused on that topic. Seeing it 100 times usually means someone is “keyword stuffing” to game search rankings.

  • The $k_1$ Knob:
    • Low $k_1$ (e.g., $0.5$): The curve flattens out very quickly. After $1$ or $2$ mentions, extra occurrences add almost no extra score.
    • High $k_1$ (e.g., $3.0$): The curve approaches a linear scale, allowing documents with high term counts to continually gain higher scores.
    • 

• Length Normaliation: Longer documents have two unfair advantages in pure keyword matching:

  • Higher raw term frequency: A 10,000-word document naturally repeats key terms far more often than a 200-word summary.
  • Multi-topic dilution: A long document might cover 20 different topics and happen to mention your query term in passing, whereas a concise 100-word abstract might be 100% focused on your query.
  • Without length normalization, long documents dominate search results simply due to sheer word count.
  •