Build Production-Ready Retrieval RAG Pipeline in LangChain | Hybrid Search (BM25), Re-ranking & HyDE

A production-ready RAG pipeline needs more than embeddings: it must reliably fetch the right chunks, even when users ask for exact numbers. A simple retrieval setup can return a convincing but wrong answer—like pulling a “4.5 billion charge” from an unrelated customer complaint policy instead of Nvidia’s financial results—because semantic search prioritizes meaning over exact matches.

The breakdown comes from “keyword blindness.” Embedding-based retrieval captures intent, but it can miss exact figures, product codes, or specific phrases that determine correctness. In the demo knowledge base, the query “Locate the section discussing the 4.5 billion charge” should land on Nvidia’s financial PDF, yet pure semantic search ranks the third chunk containing the correct figure poorly and surfaces a second chunk from customer complaint text. The result is a retrieval error that downstream generation can amplify into a fluent hallucination.

To fix recall, the pipeline switches to hybrid search using LangChain’s ensemble retriever. It combines a semantic retriever with a BM25 retriever (best match 25), which scores documents using term frequency and inverse term frequency. BM25 is designed for exact keyword matching and is widely used in search engines, making it a practical way to ensure that queries containing specific terms or numbers don’t get lost in embedding similarity. In the example, BM25 lifts the Nvidia financial chunk into the top results, but noise can still remain.

Precision then comes from reranking. Hybrid search casts a wide net to collect candidate chunks; reranking acts like an expert filter that selects the best passages among those candidates. The implementation uses Answer.AI’s Cohere-based reranker model (a “cobert type” model) inside LangChain’s contextual compression retriever. The compressor takes the ensemble retriever’s top candidates (set to three) and removes irrelevant chunks, pushing the truly relevant financial excerpts to the top.

Even with strong retrieval, real users often ask imperfect questions. The pipeline adds HyDE (hypothetical document embeddings), also called “height” in the transcript. A smaller language model generates a short hypothetical answer (2–3 sentences) to the user’s question, then appends that text to the original query. This augmentation injects concrete terms—like “financial statements” and “operating expenses”—so the retriever searches with richer signals. HyDE is especially helpful for vague queries and large document collections.

Finally, the system is wired into an end-to-end chain that produces answers with citations. A prompt instructs the model to use retrieved sources, while the chain formats the compressed, reranked chunks and feeds them into a larger chat model (the transcript mentions “Qwen3 8 billion”). In the Nvidia example query about data center revenue versus the previous quarter, the pipeline returns the correct figure (39.1 billion, up 10%) and includes the chunk source from Nvidia’s financial results.

Together, hybrid search (recall), reranking (precision), and HyDE (query strengthening)—plus source citation—turn a naive RAG setup into a system built to reduce wrong-context failures in production.