37% Better Output with 15 Lines of Code - Llama 3 8B (Ollama) & 70B (Groq)

A simple query-rewriting step inside a local RAG (retrieval-augmented generation) pipeline can materially improve answers—often by roughly 37%—even when the user’s original question is vague. The core fix: before retrieving documents, the system asks a local Llama 3 model to rewrite the user’s prompt by explicitly incorporating relevant conversation context and clarifying what the question is really asking, without changing its intent. That rewritten query then drives the document retrieval, leading to more useful context being pulled and better downstream responses.

The workflow starts with a RAG setup where questions are answered using retrieved snippets from a document collection. When the user asks something broad like “what does that mean” after asking about Llama 3’s training tokens, retrieval can fail because the query provides too little information for the retriever to find relevant passages. In the baseline run, the system retrieves no useful context for the vague follow-up, so the model has little to ground its answer.

The improved version keeps the first query unchanged, but from the second user message onward it generates a “Rewritten query.” A dedicated prompt instructs the model to (1) preserve the original intent, (2) expand and clarify the query using conversation history, (3) avoid introducing new topics, and (4) output only the rewritten query text. The implementation uses structured JSON output so the rewritten text is extracted deterministically. In code terms, the function receives the user’s original query as JSON, parses it into a Python dictionary, builds a prompt that includes the prior conversation messages, calls a local Llama 3 model (running via Ollama), extracts the rewritten query from the model response, and returns it as JSON.

Crucially, the pipeline then feeds only the rewritten query into the “get relevant context” retrieval function—skipping the original vague query for retrieval. That change is what increases the chance that the retriever finds the right document chunks.

Testing on both an 8B local Llama 3 model and a larger 70B model served by Groq, the rewritten-query approach consistently produced better retrieval and more coherent answers. For example, the follow-up “what does that mean” becomes a much more specific retrieval query describing what “15 trillion tokens” means, including token definitions and implications. The creator also reports a rough quantitative estimate: comparing two responses (one without rewriting and one with rewriting) using GPT-4 across multiple trials, the rewritten version typically scored about 30–50% better, landing around 37% in aggregate.

Beyond the rewriting trick, the project includes practical local RAG updates: switching to the “dolphin” tree Llama model, using an Ollama embeddings model for vectorization, and allowing model selection from the terminal. The takeaway is less about any single model size and more about improving retrieval quality by rewriting ambiguous follow-ups into retrieval-friendly queries using the model itself—then letting embeddings and the retriever do the heavy lifting.