Vectorless RAG - Local Financial RAG Without Vector Database | Tree-Based Indexing with Ollama

Vectorless RAG can retrieve and answer questions from structured documents without any vector database by building a tree index from the document’s own hierarchy and letting a local LLM choose which sections to use. Instead of embedding chunks and running similarity search, the system splits a document (e.g., a financial PDF converted to markdown), generates short summaries for leaf sections, then rolls those summaries up into a JSON tree. At query time, the model receives the tree’s node titles and summaries and performs “tree search” to select the most relevant nodes, using their underlying text as grounded context for the final answer.

The practical payoff is simpler infrastructure: no Postgres + PGvector setup, no embedding pipeline, and no vector-store maintenance. The approach also leans into document structure—exactly the kind of formatting found in financial filings (10-Q, 10-K) and legal documents—so the model can cite the right sections and reason across multiple levels of granularity. Because selection happens through the tree, the model can provide answers with an explicit rationale tied to why certain nodes were chosen, which can make prompt tuning more straightforward than debugging embedding-based retrieval.

The trade-offs are cost and scalability. Retrieval is slower because the model must do additional inference steps to traverse and reason over the tree rather than relying on fast vector similarity. Indexing is also expensive: summarizing every leaf (and then every parent node) requires one or more LLM calls per section, so costs rise with document size and number of documents. The method doesn’t scale well when the model’s context window is limited (the transcript cites a 30k context window as a pain point). Summaries-of-summaries can help, but the “basic implementation” still struggles as datasets grow.

When to use which approach comes down to document count, structure, and cost. For small, well-structured local RAG workloads—especially where the LLM can handle the inference overhead—vectorless tree-based indexing can perform well. For thousands of documents, classical RAG with embeddings is likely faster and more scalable because vector search remains efficient, and systems can add hybrid search (keywords + embeddings) to improve chunk selection.

The implementation shown runs entirely locally using an LLM from the transcript’s example: “geometry, the 4 billion parameter version.” The tree nodes store a title, note ID, content, and an LLM-generated summary, plus children pointers. The indexing step uses LangChain’s markdown header text splitter to create hierarchical sections, then performs bottom-up summarization via an iterative traversal (a stack-based approach). Query tests include questions like “What was NVDS total revenue and earnings per share in Q3?” and “What partnerships did Nvidia announce for AI infrastructure?” The system retrieves the relevant sections and produces answers grounded in the extracted filing text. For the partnerships question, the most relevant node is identified as “00005 data center,” which cites a strategic partnership with OpenAI to deploy at least 10 GW of Nvidia systems, along with Microsoft, Oracle, and XAI. For Q4 outlook, the retrieved outlook node yields revenue expectations of 65 billion ± 2% and gross margins of 74.8% (GAP) and 75% (non-GAAP), matching the document’s figures.