Text Splitters in LangChain | Generative AI using LangChain | Video 11 | CampusX

Text splitting is the practical step of breaking large documents—PDFs, articles, HTML pages, books—into smaller chunks that an LLM can handle reliably. Instead of feeding a massive input and accepting weaker answers, chunking improves output quality across common RAG tasks like semantic search, embeddings, and summarization. The core reason is hard limits: most LLMs and embedding pipelines can only process up to a fixed context length (e.g., tens of thousands of tokens). When documents exceed that threshold, the system either fails to ingest everything or produces lower-quality, less faithful results.

Beyond context limits, chunking helps because downstream steps work better with focused text windows. For embeddings, a single embedding for a very large blob often fails to capture multiple distinct topics’ semantic meaning. Splitting by natural boundaries (like paragraphs or sentences) yields embeddings that represent each topic more accurately. That improvement carries into semantic search: when queries are compared against embeddings built from smaller, topic-coherent chunks, similarity scores become more precise, so the retrieved passages match the user’s intent more closely. Summarization also benefits; large inputs can cause “drift,” where the model starts talking about nearby but irrelevant content or even introduces details not present in the source. Smaller chunks reduce that risk.

Chunking also optimizes compute. Processing smaller pieces typically uses less memory, stores fewer tokens per unit, and enables more parallel execution—important when building production RAG pipelines.

After establishing why chunking matters, the lesson moves into how to do it in LangChain, focusing on four chunking strategies. The first is length-based splitting: cut text into fixed-size segments (by characters or tokens). It’s fast and simple, but it ignores grammar, sentence boundaries, and semantics—so words or sentences can be cut midstream, and a topic can get split across chunks, weakening embeddings.

To mitigate abrupt cuts, the workflow introduces chunk overlap: adjacent chunks repeat a small number of characters so context isn’t lost at boundaries. Overlap creates a tradeoff: more overlap means more chunks and more computation, but better continuity for embedding and retrieval.

The second major approach is structure-based splitting using RecursiveCharacterTextSplitter. It tries separators in a hierarchy—paragraphs first, then sentences, then words, and finally characters—recursively until each chunk fits the target size. This preserves linguistic structure better than pure length-based splitting, especially when chunk sizes are chosen sensibly.

The third approach extends the same recursive idea to non-plain-text formats. For code and Markdown, separators change to match language constructs (e.g., class/function blocks, Markdown headings/lists). The goal is still coherent chunks, but the “boundaries” come from syntax rather than paragraphs.

The fourth approach is semantic meaning-based splitting, implemented via LangChain’s experimental SemanticChunker. It embeds sentences (using an embedding model such as OpenAI embeddings), then detects topic shifts by measuring changes in similarity across a sliding window. Breakpoints are chosen using thresholds such as standard deviation (and other statistical options like percentile or interquartile ranges). Results can be sensitive to threshold settings, and the method is described as promising but still experimental.

Overall, RecursiveCharacterTextSplitter emerges as the most reliable default for typical RAG pipelines, while semantic chunking remains an experimental option for cases where topic boundaries don’t align cleanly with length or structure.