LangChain Retrieval QA Over Multiple Files with ChromaDB

LangChain retrieval QA becomes practical at scale once the document embeddings live in a persistent ChromaDB vector store on disk. Instead of rebuilding an in-memory index every time the app starts, the workflow loads many text files from a folder, chunks them, embeds the chunks, and writes the resulting vector index into a persisted directory (named “DB”). That persistence matters most when the corpus grows to hundreds or thousands of long documents, where re-embedding on every launch would be slow and expensive.

The setup starts by pulling multiple files using a simple folder-based loader pattern: a directory is scanned with a glob like “*.text” (and the loader can be swapped for other formats such as PDFs or Markdown by changing the file extension and loader type). After loading, the documents are split into chunks—small units that fit within embedding and LLM context limits. With chunks ready, the code initializes embeddings using OpenAI for both the language model and the embedding model. Those embeddings are then used to create a Chroma vector store from the chunked documents, with a persistence directory so the index is saved to disk. Once persisted, the original embedding step can be skipped on subsequent runs by reloading the vector store from the same directory.

After the vector database is in place, the system turns it into a retriever configured for similarity search. A query is issued, the retriever returns the top matching chunks (the example uses k=2, with guidance that k≈5 often works well when answering from multiple sources). Those retrieved chunks become the “context” fed into a RetrievalQA chain, which combines the context with the user’s question and asks the LLM to answer strictly using the provided material. The chain is configured to return the source documents as well, enabling citation-style outputs rather than ungrounded answers.

Concrete examples show the retrieval-and-answer loop working: questions about Pendo’s fundraising return a “30 million” figure with the relevant source chunks; a query about “news about Pando” yields both an answer and the underlying documents, including details like the round type and usage; and questions like “what is generative AI?” or “who is CMA?” pull answers from different articles based on semantic similarity. The prompt template reinforces the guardrail: if the answer isn’t in the retrieved context, the model should say it doesn’t know.

Finally, the notebook demonstrates swapping the LLM backend to the GPT-3.5-turbo API (via LangChain’s chat prompt structure with system and human messages). The retrieval and ChromaDB wiring stays the same; only the prompt formatting changes to match the chat model interface. The result is a reusable, disk-backed retrieval QA pipeline with grounded answers and source traceability—positioning it for future upgrades like Pinecone deployment or local embeddings for the lookup stage.