How Token Caching Reduces API Costs

What gets cached

Provider prompt caching can discount input that matches reusable prefixes or explicitly stored context. Eligibility, minimum sizes, retention, and pricing differ by provider and model.

A nominal cache discount does not apply to every input token. Separate eligible input, measured hits, misses, writes, and any storage charges.

Calculate the blended rate

Multiply the cached share by the cached-input price and the remaining share by the standard-input price. Add output cost separately. A calculator’s cache-rate input should represent successful billed cache hits, not the share you hope to reuse.

Design prompts for reuse

Place stable instructions and reference context before request-specific content when supported. Avoid incidental changes in reusable prefixes. Track hit rates by feature and model so optimizations are evidence-based.

Do not optimize cost alone

Caching can improve latency as well as price, but large static prompts may still be wasteful. First remove unnecessary context, then cache the stable material that remains. Recheck assumptions whenever provider rules change.