Claude Prompt Caching: Did Anthropic Create a Better Alternative to RAG?

Anthropic’s new Prompt Caching for Claude is designed to cut both cost and latency by reusing frequently used prompt context across API calls—an approach that could rival parts of RAG when the “retrieval” content is stable and repetitive. In public beta, developers can mark a prompt prefix for caching; when a later request shares that same prefix, the API serves the cached version instead of reprocessing the entire input. Anthropic positions the payoff as up to 90% lower cost and up to 85% lower latency for long prompts, with a cache lifetime that refreshes on use and expires after a short window (described as about five minutes).

The practical setup centers on a “cache control” block placed near the beginning of the prompt. In the example walkthrough, a large text—Harry Potter and the Chamber of Secrets—is loaded from a file, and the system is instructed to cache the book content using the cache control parameter. The caching logic is straightforward: the API checks whether the prompt prefix already exists from a recently processed request; if it does, it reuses the cached prefix, reducing processing time and cost. If it doesn’t, the full prompt is processed once and the prefix is cached for future calls. The cache is not manually clearable in the current beta; it expires and refreshes on use, which matters for testing and for production scheduling.

Pricing details reinforce why caching targets long, repeated context. Cached input is billed at a higher per-token rate than base input tokens, but the overall economics improve when the cached portion is reused enough times. In the transcript’s example numbers for Claude 3.5 Sonnet, base input is $3 per million tokens, while cached input is $3.75 per million tokens. Yet when the cached content is reused, the effective cost of the repeated input drops dramatically—illustrated as roughly 10% of the original input cost for the reused portion—while output pricing stays the same.

The walkthrough then tests two concrete use cases. First, it summarizes the Harry Potter book and measures timing: the initial run takes about 21 seconds and caches roughly 138,000 input tokens. A second run of the same request drops to around 8 seconds, with no additional cached input tokens—showing the latency benefit from reusing the cached prefix. A “needle-in-a-haystack” style query (asking about a specific sentence buried in the book) is then run after the cache window has likely expired; surprisingly, the cached token count appears again as ~138,000, with much faster runtimes (around 2–3 seconds), though the transcript notes some confusion about the exact behavior.

Second, the transcript demonstrates caching a smaller but still structured prompt: a dataset of 40 example YouTube comments used to mimic a specific response style. The cached portion is about 3,600 tokens, and the measured runtime is roughly 3.2 seconds both times, suggesting limited speedup for smaller prompts but potential cost savings at larger scales (e.g., hundreds of examples). Overall, Prompt Caching is framed as a promising alternative or complement to RAG when the “context” is long and stable, especially as Anthropic’s slower models (like Claude 3.5 Opus, mentioned as upcoming) could benefit most from reduced latency through reuse.