What it is
Prompt caching lets an application mark the start of a request, such as a long system prompt or a set of tool definitions, as cacheable. The first request writes that prefix to a cache. Later requests that begin with the same prefix read it back instead of processing it again, which is faster and cheaper.
Cache reads are billed at a fraction of the normal input price, while the first write costs a little more. A cached prefix lives for a few minutes by default and is refreshed each time it is used, so a busy endpoint keeps it warm. It works best when the instructions stay the same and only the question changes.
How I use it
- The agent on mervinc.com caches its system prompt and tool definitions, so after the first question they cost a tenth as much.
- Every answer on the ask page shows tokens in and out, how many came from cache, and the cost of that answer.
- Cache reads and writes are priced separately in the agent's cost calculation, so the cents shown are the real ones.
- Saved answers in DynamoDB go further: a repeat question is answered without calling the model at all.
Prompt caching is worth turning on whenever an agent sends the same long instructions and tool list with every question. That is most agents. It is one line of configuration, and on a public page where strangers ask questions on my budget, it is the difference between a cost I notice and one I do not.
Questions
How much does prompt caching save?
Cached input is billed at a tenth of the normal input price. On my agent, the system prompt and tools cost a tenth as much after the first question.
How do you keep AI costs predictable on a public site?
Cache the instructions, save good answers so repeats cost nothing, and count per-visitor and site-wide daily caps before the model is called.