How to use BGE Embeddings for LangChain and RAG

BGE embeddings from the Beijing Academy of AI have surged to the top of major embedding benchmarks while dramatically shrinking model size—making them a practical upgrade for retrieval-augmented generation (RAG) pipelines built with LangChain and Chroma. The core takeaway is straightforward: better benchmark performance isn’t the only win. The smaller BGE models cut embedding latency, reduce RAM/VRAM requirements, and can even be run on a CPU, which changes the cost and operational friction of building and maintaining a vector database.

The transcript starts by challenging a common confusion in the embedding ecosystem: using an OpenAI text model doesn’t require using OpenAI embeddings. It argues against long-term reliance on OpenAI embeddings for production RAG systems, mainly due to vendor lock-in. Once a large corpus is embedded with a proprietary provider, switching later forces a full re-embedding of everything—an expensive and time-consuming reset. There’s also a risk of future deprecation: as newer embedding models arrive, older OpenAI embeddings may be retired, again requiring re-embedding. OpenAI embeddings are framed as acceptable for quick experiments—testing an idea or validating a prototype—but not ideal for major projects that need portability and longevity.

From there, the focus shifts to BGE embeddings. These models come with separate English and Chinese embedding variants, and the team is also working on multilingual embeddings that aren’t released yet. In the benchmark context referenced (hosted by Hugging Face via the Massive Text Embedding Benchmark leaderboard), BGE models have rapidly climbed in just the past few days. A key comparison is size: the previously favored instructor XL embeddings are roughly under 5 GB, while the BGE base English model is about a tenth that size. A larger BGE variant is still far smaller than instructor XL (just over 1 GB) and uses bigger embedding dimensions.

The practical demonstration uses the BGE base English embedding model with LangChain and a Chroma vector store. The pipeline stays largely the same—using a Llama 270B model hosted on Together API for retrieval QA—while swapping the embedding backend. The transcript specifies cosine similarity by normalizing embeddings (“normalize the embeddings as true”) and then plugging the embedding function into Chroma in the same way as before.

The operational impact is concrete: embedding roughly 1,000 text chunks drops from minutes with instructor XL to about 35 seconds with BGE base en on a T4. Retrieval quality is described as at least comparable, with the system correctly finding the right outputs in the tested cases. Some answers include odd or inconsistent extra text, suggesting that a second pass (output cleanup) could improve final responses.

Overall, the “big win” is less about a dramatic jump in retrieval accuracy and more about efficiency: smaller models mean faster embedding time, lower memory demands, and the possibility of CPU-based inference. The transcript ends by recommending close attention to BGE’s upcoming multilingual embeddings, anticipating improvements for multilingual RAG use cases.