Augmented Language Models (LLM Bootcamp)

Augmented language models hinge on a simple constraint: modern LLMs are strong at language and instruction-following, but they lack up-to-date world knowledge, access to a company’s private data, and reliable performance on harder tasks like math-heavy reasoning. The practical fix isn’t retraining the model for every new need—it’s giving the model the right external “tools” and “data” at inference time. The core idea is to treat an LLM less like a self-contained oracle and more like a reasoning engine that needs a curated context to answer real questions.

The starting point is retrieval augmentation, which reframes “stuffing context into the prompt” as an information-retrieval problem. Instead of manually deciding which user or document snippets to include, systems search an external corpus for the most relevant items and then inject those results into the prompt. This matters immediately in multi-user settings: rules like “include the most recent users” or “include users mentioned in the query” break down when the relationship between a question and the right data is too complex to encode with simple logic. Retrieval turns that selection step into something searchable and measurable.

Traditional search relies on inverted indexes and word-level heuristics such as Boolean filtering and ranking methods like BM25. Those approaches work well for exact or unambiguous keyword matches, but they miss semantic meaning—so ambiguous queries can return documents about the wrong sense of a word. Embeddings shift the approach by representing text (and other modalities) as dense vectors so semantically similar items land near each other in vector space. The retrieval pipeline then becomes: embed the query, find nearest neighbors among embedded documents, and pass the top matches into the LLM prompt.

The lecture emphasizes that “embedding databases” are often unnecessary at small scale. For fewer than roughly 100,000 vectors, simple nearest-neighbor search with NumPy can be enough. As scale grows, approximate nearest neighbor methods speed up lookup by using specialized index structures (e.g., HNSW-based approaches) that trade a bit of accuracy for speed. But production systems need more than a fast index: they must handle metadata, filtering, embedding management, and reliable ingestion/update workflows. The practical recommendation is to start with whatever database already powers the application—many support vector search via extensions or built-in features—then graduate to dedicated vector databases when you need richer capabilities.

Beyond nearest-neighbor retrieval, the biggest limitation is context window size. If only a few documents fit, the system can fail when the answer isn’t retrieved into those top slots. One workaround is “chains”: use an LLM to re-rank or select among a larger candidate set before the final answer prompt. Chains also generalize to other patterns like hypothetical document embeddings (invent a document that might contain the answer), map-reduce style summarization across large corpora, and multi-step workflows orchestrated with frameworks such as LangChain.

Finally, tools broaden augmentation beyond search. Rather than only retrieving documents, LLMs can call external systems—calculators, SQL databases, APIs—either through developer-defined chains or through plugin-style interfaces where the model decides when tool use is helpful. The takeaway is a hierarchy of augmentation strategies: start with retrieval and heuristics, move to chains for more complex context-building and token-limit workarounds, and use tools/plugins when the model needs to interact with external capabilities or live data.