Prompt caching cuts AI costs, and it keeps your data in memory

An initial 3,000-token prompt followed by five rounds of conversation, each adding 1,000 tokens. Without caching, that session costs 30 units. With a warm cache on a 5-minute TTL, it costs 12.2. With a cold one, it costs 37.5, more than if you'd never turned caching on. Same prompt, same model, and the only variable is whether the prefix matched. A prefix is the unchanging beginning of a request, like the instructions and background material, that the AI can reuse instead of reprocessing each time.

Based on Anthropic’s pricing multipliers (cache read 0.1x, 5-minute write 1.25x), counting input tokens only.

The example spread is reason enough to learn prompt caching. The cache also holds whatever your people and pipelines put in the prompt, and that part gets far less attention.

What is prompt caching and how does it work

When a model reads a prompt, it computes key-value (KV) states for every token so it can refer back to them while it writes the answer. Normally those states get discarded when the response ends. Prompt caching keeps them. The cache stores the KV tensors, not the raw tokens, and a later request that opens with the identical prefix reuses that saved state and only does fresh work on whatever is new.

Matching is exact and prefix-based. Providers compare the start of your request against what they've recently cached, in this order: tools, then system prompt, then messages. Change a date in the system prompt and everything after it misses. A cache write commits the conversation so far for a set time to live (TTL), and every hit resets that clock, so an active session stays warm on its own.

Five ways prompt caching stops saving money

Volatile values near the top. A current date, a user name, or a request ID someone added for logging turns every call into a write. One team moved a single volatile field to the end of the prompt, after the cached part, and the reads finally started landing.

Pauses longer than the TTL. Step away from a 5-minute session for a meeting and the next turn pays full price for the entire conversation. The all-miss row in the table above is what that looks like.

Compaction. Every compaction is a cold start, one full-price turn on a summarized history the provider has never seen. Run /compact before a break while the cache is still warm. Once it has expired, compacting costs more than just continuing.

Mid-session changes. Switching the model, the effort level, plugins, or MCP servers can alter the system prompt and invalidate the cache. Check the provider's docs before you change settings partway through a long task.

Scale and routing. At high volume, requests land on machines without your cached entry. Bedrock's automatic prefix check only looks back about 20 content blocks from a checkpoint, so a long static section needs more than one.

How to check that prompt caching is working

Compare cache write tokens to cache read tokens for each feature. If writes track your request count, nothing is being reused. Confirm that cache_read_input_tokens is above zero on the first turn after an idle period. On Bedrock's Converse API, read cacheReadInputTokens and cacheWriteInputTokens, and remember that inputTokens then counts only uncached tokens, so total input is inputTokens plus both cache fields. On GPT-5.6 and later, a response with cached_tokens above zero and cache_write_tokens at zero means a full match.

To keep a large session warm, send a small useful request every 20 to 25 minutes. An inactivity-based timer works better than a fixed ping, because real work resets it anyway.

Prompt caching and sensitive data

The golden rule of caching is static content first and the user's question last. Read that rule as a data inventory. The cacheable parts are tool definitions, system prompts, few-shot examples, large reference documents, and older conversation turns. In a RAG setup, the reference documents are the same files injected into every request. In a chat, the conversation history is whatever people typed.

Let’s say an employee submits a system event entry including an internal user ID, a timestamp, and a transaction tracking code. In a cached session, that text rides in the prefix on every following turn, and each turn refreshes the clock on it.

Providers do address this. Caches aren't shared across organizations, even for byte-identical prompts. Anthropic and OpenAI both describe caching as eligible for zero data retention: raw prompt text isn't stored at rest, and only the computed KV state sits in memory, or briefly offloaded for extended retention, until the TTL ends. Google's explicit caching supports VPC service controls on its enterprise platform. Those are real controls.

They also leave a window, and you choose how long it is. That's 5 minutes to 1 hour on Anthropic models, 30 minutes to 12 hours on OpenAI's GPT-5.6 and later, and up to 24 hours on earlier OpenAI models with extended retention. Heartbeat pings stretch it for as long as someone keeps sending them. Read each provider's current data-handling terms before you cache anything sensitive.

Cost tuning pulls the wrong way too. A larger stable prefix caches better, so a team chasing that 50% repetition mark has a reason to load more documents into it. The savings math says nothing about what those documents contain.

Prompt injection raises the stakes. Indirect injection hides instructions in files an assistant is likely to read, like a routine PDF or a calendar invite, and the EchoLeak exploit carried a CVSS score of 9.3. A poisoned file in a cached reference set gets re-read with every request until someone pulls it.

Shadow AI adds the part IT can't see. Most teams see only a small share of the AI tools their people use, from Claude.ai and Perplexity to Gemini and browser extensions. Banning them pushes employees to personal devices, where security tools can't reach. Security leads are moving to group-specific guardrails instead: scanners for leaked API keys and hard-coded secrets for engineering, and blocked uploads or tokenized credentials for legal and finance.

Tokenization is one answer on the input side. A gateway replaces a patient name with a logical token like [PERSON_01] in under 300 milliseconds, so the model still follows the narrative without seeing the identity. A scanner can only mask what it recognizes, though, which makes classification the step that has to come first.

Auditors are asking for the same trail. SOX, PCI DSS 4.0, and Article 25 of the EU AI Act all point toward logging and traceability for AI interactions, including a unique request ID per prompt and a recorded reason for every block. A cache is one more place the data lived, and the question of how long goes on the list.

Before you turn on prompt caching

• Put static content first and the user's question last.
• Decide which sources may enter a cached prefix, and keep customer records, credentials, and regulated data out.
• Choose the shortest TTL that still produces hits.
• Track cache reads against cache writes for each feature.
• Audit what your reference documents contain before they join the prompt.

Classify what goes into the prompt before it gets cached

Every control above depends on knowing which files are sensitive. Netwrix Data Classification answers that at the source. It continuously discovers and classifies sensitive and regulated data across Windows file servers, NetApp, Dell EMC, Nutanix Files, SharePoint, Microsoft 365, Exchange, SQL Server, Oracle, MySQL, Box, Dropbox, and Google Drive. Predefined rules cover GDPR, HIPAA, and PCI DSS, and custom taxonomies handle everything else.

Classification tags embedded in files feed DLP and IRM tools, so a document marked sensitive can be kept out of an AI workflow before anyone loads it into a prefix. ROT detection finds redundant, obsolete, and trivial files, which cuts the stale content sitting in your reference sets. Automated remediation moves sensitive data to secure locations, removes excessive permissions, and redacts confidential content.