100% Local CAG with Qwen3, Ollama and LangChain - AI Chatbot for Your Private Documents

Cache-augmented generation (CAG) is presented as a simpler alternative to retrieval-augmented generation (RAG) for private-document chat: instead of searching a database at question time, the system preloads your documents into the model’s context window and reuses that cached prefix during the conversation. The payoff is speed and cost efficiency—prompt caching can make subsequent turns cheaper—while keeping everything local when paired with open models and local inference.

The core idea traces back to a paper on “don’t do RAG when cache augmented generation is all you need for knowledge tasks.” The transcript argues the approach is increasingly practical because modern large language models can accept very large context windows—citing examples like Gemini 2.5, OpenAI o3, and Qwen3.7 with 100K to even 1M tokens. In that setup, a knowledge base (PDFs, text, or URLs) is converted into content that can be inserted up front, then cached by the model provider or local runtime. The speaker draws parallels to how commercial APIs do prompt caching by caching a prefix of tokens and reusing it across later messages.

A key comparison is made between CAG and RAG. RAG depends heavily on retriever quality (chunking, indexing, and whether the right passages are retrieved). CAG depends on the model’s ability to handle long contexts and “search effectively” within them. That’s why the transcript emphasizes evaluation beyond context length alone, pointing to a benchmark for long-context comprehension (FictionBench) that tests whether models can answer questions that reference details appearing early in long documents. The results mentioned highlight that some models handle context better than others, and that even open models like Qwen (the one used in the build) may lag behind top closed models such as Gemini 2.5 Pro or OpenAI o3.

To decide when CAG is appropriate, the transcript offers heuristics: first, check whether the document content fits within the model’s context window with room for chat history. If the knowledge is mostly static—like a fixed FAQ plus a few documents—CAG can avoid the overhead of retrieval. If the knowledge changes dynamically per user (e.g., purchase history), RAG-style retrieval may be more suitable because the injected context must vary.

The implementation is then built as a fully local chatbot using Qwen3 via Ollama and LangChain, with a Streamlit UI. Documents can be uploaded as PDF, text, or Markdown, or added by URL. Dockling is used to convert PDFs and web content into Markdown, and the app stores each converted “knowledge source” in memory as a structured object (ID, name/type, and content). The chatbot constructs a prompt template that injects the cached document context plus the user question, then streams the model output. For Qwen3 specifically, the transcript notes special “thinking” tokens and renders the model’s intermediate reasoning in an expandable UI section.

A small test demonstrates how Ollama’s internal caching behaves by enabling debug logging and observing cache slot usage across turns. In the final demo, the system answers questions about an Apple financial statement PDF and cites numbers from tables, with responses appearing fast after the initial cached context is established. The result is a local, end-to-end CAG workflow that trades retrieval complexity for long-context caching—when the documents are static and the model can reliably reason over large inputs.