Understanding OpenAI Prompt Caching: Why Does Prefix Order Matter?

Large prompts often repeat the same instructions, examples, tool definitions, or reference material while only the final user request changes.

The model still needs the entire context, but it may not need to recompute the unchanged beginning every time.

That is the idea behind prompt caching.

This project makes OpenAI's automatic prefix caching visible with a long fictional travel-support policy and a series of changing customer cases.

First: What Is Prompt Caching?

Prompt caching reuses model work for an unchanged prefix.

Request 1
[long stable policy][case A]
         ↓
process stable prefix + changing suffix

Request 2
[same stable policy][case B]
         ↓
reuse eligible prefix + process changing suffix

The response still considers the complete prompt. Caching changes how repeated input is processed and billed; it does not remove the cached instructions from the model's context.

A Prefix Is an Ordering Constraint

The reusable content must appear first.

GOOD FOR REUSE

stable policy
stable examples
stable tool schemas
changing customer case

Putting changing content at the beginning breaks the shared prefix early:

POOR FOR REUSE

case A + stable policy
case B + stable policy
  ↑
prefix diverges immediately

The project tests both arrangements instead of assuming that repeated text anywhere in the request is enough.

The OpenAI Cache Is Automatic

The demo does not create a cache resource. It makes a normal Responses API request:

response = client.responses.create(
    model=model,
    instructions=policy,
    input=case,
    prompt_cache_key="northstar-travel-policy-demo",
    max_output_tokens=180,
    store=False,
)

Supported OpenAI models automatically cache eligible prompt prefixes. A later matching request may reuse the cached model state.

There is no create_cache() call and no cache identifier returned for the application to persist.

What Is Actually Cached?

Conceptually, transformer inference builds key-value states while processing earlier tokens.

prompt tokens
    ↓
attention key-value state
    ↓
later tokens and output generation

OpenAI's documentation describes the cache as preserving reusable KV tensors rather than storing a second user-visible copy of the prompt tokens.

The cache is an inference optimization. It is different from the key-value memory article, where the application stores typed business facts in SQLite.

PROMPT CACHE                      APPLICATION KV MEMORY

model computation state          durable business data
matched by token prefix          addressed by namespace/entity/key
managed by provider              managed by application
short-lived reuse                explicit lifecycle and persistence

Why the Demo Uses instructions First

The stable travel policy is passed through instructions, and the changing customer case is passed through input.

instructions=policy
input=case

That request shape gives every case the same long beginning and a small variable suffix.

The policy contains 75 fictional support rules and is intentionally long enough to create a meaningful cacheable prefix for the selected model.

What prompt_cache_key Does

The demo defaults to gpt-4.1-mini, an earlier model family, and supplies one stable cache key:

prompt_cache_key="northstar-travel-policy-demo"

For models before GPT-5.6, OpenAI documents this key as a routing aid for requests that share a reusable prefix. It helps related calls reach compatible cache state.

It cannot force a cache hit.

stable key + stable prefix
→ improves routing opportunity
→ does not guarantee reuse

On GPT-5.6 and later, OpenAI handles cache routing automatically. A key is optional and can instead separate cache accounting across customers or workspaces.

store=False Is a Different Feature

The request also uses:

store=False

That controls Responses API response storage. It does not disable prompt caching.

These settings affect different layers:

store=False
→ response-storage behavior

prompt caching
→ reusable inference state for matching prefixes

Conflating the two can lead to a false conclusion that a non-stored response must also be uncached.

Measure Cached Tokens, Not a Fast Feeling

The script reads usage returned by the API:

usage = response.usage
details = usage.input_tokens_details

cached_tokens = details.cached_tokens if details else 0

cached_tokens is a subset of input_tokens.

input tokens = cached input + uncached input

The summarizer therefore computes:

uncached = total_input - total_cached
hit_rate = total_cached / total_input

Latency is printed as an observation, but a few response times do not prove a cache hit. Routing, load, output length, and network variation also affect elapsed time.

The authoritative signal is the usage field.

The Three Experimental Phases

The demo makes twelve generation calls across three layouts.

Phase 1: Varied Prefix

Each case gets a unique line before the shared policy:

variant = f"Unique policy edition for case {index}: {case}\n{policy}"

The beginning changes on every request, reducing the opportunity for a long match.

This is a cold-prefix illustration. It is not a server-side switch that disables caching.

Phase 2: Stable Prefix

Every request sends the exact same policy first and a different case last.

policy A + case 1
policy A + case 2
policy A + case 3
policy A + case 4
policy A + case 5

The first request can establish eligible cache state. Later requests can report cached tokens if routing and lifetime conditions align.

Phase 3: Changed Prefix

The project changes the policy revision near the beginning:

changed = POLICY.replace(
    "revision 2026-09-A",
    "revision 2026-09-B",
    1,
)

The old prefix no longer matches beyond that early edit. A later request using revision B may reuse state created by the first revision-B request.

Controlled Output From the Real Measurement Code

I ran run_phase() and summarize() from the real project against an offline client fixture. The fixture supplied deterministic usage metadata so the aggregation and prefix experiment could be reproduced without making billable API calls.

Phase 1: varied prefix
  1: 2300 input, 0 cached, 40 output, 0.25s
  2: 2300 input, 0 cached, 40 output, 0.25s
  3: 2300 input, 0 cached, 40 output, 0.25s
  4: 2300 input, 0 cached, 40 output, 0.25s
  5: 2300 input, 0 cached, 40 output, 0.25s
  cached share 0.0%; mean latency 0.25s

Phase 2: stable prefix
  1: 2300 input, 0 cached, 40 output, 0.25s
  2: 2300 input, 2048 cached, 40 output, 0.25s
  3: 2300 input, 2048 cached, 40 output, 0.25s
  4: 2300 input, 2048 cached, 40 output, 0.25s
  5: 2300 input, 2048 cached, 40 output, 0.25s
  cached share 71.2%; mean latency 0.25s

Phase 3: changed prefix
  1: 2300 input, 0 cached, 40 output, 0.25s
  2: 2300 input, 2048 cached, 40 output, 0.25s
  cached share 44.5%; mean latency 0.25s

The totals were:

varied  input=11500 cached=0    uncached=11500 output=200
stable  input=11500 cached=8192 uncached=3308  output=200
changed input=4600  cached=2048 uncached=2552  output=80

These numbers demonstrate the script's accounting behavior. They are controlled fixture values, not claims about a live OpenAI cache or current pricing.

What a Live Run Can and Cannot Prove

A live run observes provider behavior at one moment.

cached_tokens > 0
→ this request reused an eligible cached prefix

cached_tokens = 0
→ no measured reuse for this request

A zero does not necessarily mean the layout is wrong. Possible causes include an ineligible prefix length, cache expiry, model or setting changes, routing, changed tools, changed content ordering, or a first request that is establishing cache state.

The demo prints a diagnostic message when the stable phase reports no hits instead of presenting latency as evidence.

Cache-Affecting Inputs Extend Beyond Text

The effective rendered context may include:

• developer instructions
• conversation history
• tool definitions and ordering
• structured-output schemas
• images, documents, or supported audio
• model and request settings

Changing those elements can change the reusable prefix. Stable textual instructions alone are insufficient if the tool list is reordered on every request.

Cost Needs Current Model Pricing

The script intentionally avoids hard-coded dollar estimates.

Cached-input, uncached-input, and cache-write rates vary by model and can change. The useful measurements are:

input_tokens
cached_tokens
cache_write_tokens when available
output_tokens
request latency

Join those observed counts with the current price for the exact model when calculating cost.

What the Tests Prove

The OpenAI portion has two offline tests. They verify that:

• the stable policy is sent as instructions before the changing case
• store=False remains present
• the stable prompt_cache_key is supplied
• returned cached-token usage is captured
• cached tokens are subtracted correctly from total input

The combined OpenAI and Gemini project completed with:

Ran 5 tests in 0.001s
OK

The tests prove request construction and accounting. Only a live API call can observe a real provider cache hit.

What I Took Away

Prompt caching is primarily a prompt-layout and measurement problem.

put stable content first
append changing content last
keep cache-affecting settings stable
read cached-token usage
expect cache misses

The provider owns cache placement and lifetime. The application owns the ordering that creates a reusable prefix and the telemetry that proves whether reuse occurred.

Source Code

The complete project includes the long fictional policy, twelve-call experiment, usage aggregation, cache-key routing example, diagnostics, Gemini comparison, and offline tests.

References