Build Hour: Prompt Caching

Prompt caching is positioned as a straightforward way to cut both latency and cost in OpenAI-powered applications by reusing computation whenever new requests share the same input prefix. The core mechanism is compute reuse: when multiple requests begin with identical content (the “prefix”), the system skips reprocessing tokens it has already handled and only runs inference for the new portion. That matters most for long prompts, high-volume workloads, and agentic systems where conversations and tool traces repeat large chunks turn after turn.

Caching eligibility starts at 1,024 tokens. Prompts shorter than that don’t qualify, while caching begins in 128-token blocks once the request reaches the threshold. A key constraint is contiguity and order: cache hits require the exact same prefix content in the same sequence. OpenAI describes this as implicit prompt caching across modalities—text, images, audio, and more—triggered automatically on the Responses API (and also on certain other endpoints mentioned), without code changes.

The financial impact is tied to cache hit rate. OpenAI introduced steep discounts for cached input tokens across model families (with larger discounts for newer families), and the session highlights that audio caching on the speech-to-speech model can reach nearly 99% discount on cached audio tokens. Latency gains depend on prompt length: a benchmark with variable-length inputs (from 1,024 up to 200,000 tokens) shows cached requests keeping time-to-first-token relatively stable, while uncached requests slow sharply as inputs grow. The practical takeaway is that the longer the input, the more caching helps time-to-first-token.

Under the hood, OpenAI hashes the first 256 tokens of the prefix and uses that hash to route requests to the right backend engine. Engines have capacity limits (about 15 requests per minute), so traffic distribution can trade off cache hit rate versus overall system health. When a request arrives, the engine checks how many subsequent 128-token chunks match what it has already cached; inference runs only from the first mismatched token onward. After generating output, the system updates the cache so future requests can reuse the intermediate key-value tensors (not raw tokens or media).

Several developer tactics are presented to maximize cache hits. The optional prompt cache key is emphasized as a way to improve routing locality when many requests share only part of the prefix; it hashes the prefix plus the key so related requests land on the same machine. Demos show that accidentally breaking the cache—such as inserting timestamps or dynamic whitespace early in the prompt—can drive cache hit rates to 0%, while using prompt cache keys can materially raise hit rates (and reduce input token cost) even when latency changes are small.

Context engineering is treated as a balancing act: shaping prompts dynamically can improve task quality, but it can invalidate cached prefixes. Newer compaction options (including server-side compaction and a responses/compact endpoint) help manage long-running contexts by summarizing or trimming history before it grows too large. For real-time speech-to-speech use, a retention ratio parameter is described as a way to avoid truncating every turn and instead make larger cuts less frequently.

The session also adds operational guidance: use the Responses API with reasoning models to preserve hidden chain-of-thought tokens that otherwise reduce cache reuse; consider flex processing for asynchronous workloads; and use allowed tools to change tool access without invalidating the cache. A customer spotlight from Warp frames prompt caching as essential for agentic loops—where system prompts, tool definitions, and earlier turns repeat—then describes Warp’s “scopes” approach (global, user, task) and its use of prompt cache keys to more than double cache hit rates. The closing message is that prompt caching is framed as having no direct intelligence trade-off when prefixes remain identical; the only real trade-off is architectural—how to structure prompts and context so caching remains effective.