LLM Response Caching to Save Costs: A How-To

If you're calling an LLM API and paying per token, response caching is the single highest-leverage optimization you can make. Instead of sending the same (or a similar) prompt to the model repeatedly and paying for it every time, you store the response once and serve it from cache on subsequent matching requests. For applications with repetitive queries — support bots, documentation search, classification tasks, FAQ generation — this can cut LLM spend by 30–80% with no change to output quality.

This article covers how to actually build LLM response caching: what to cache, how to generate cache keys, how long to keep entries, and the tradeoffs between exact-match and semantic caching. It's a practical implementation guide, not a theory piece.

Why response caching works for LLM costs

LLM pricing is per-token, and tokens are expensive relative to a cache lookup. If 20% of your traffic is users asking "What's your refund policy?" in slightly different wording, you're paying full inference cost for the same answer over and over. A cache turns repeat queries into near-zero-cost lookups.

The catch is that LLM inputs are rarely byte-identical across requests. Caching an LLM response well means solving two problems: deciding what counts as "the same request," and deciding how long a cached answer stays valid.

Step 1: Identify cacheable request patterns

Not every LLM call is a good caching candidate. Look for:

• High repetition: FAQ-style questions, product descriptions, common classification inputs.
• Low personalization: Responses that don't depend on user-specific context (if they do, include that context in the cache key).
• Tolerance for slight staleness: If the underlying data changes rarely, a cached answer from an hour ago is still correct.

Avoid caching anything with live data dependencies (current prices, today's weather, account balances) unless you invalidate aggressively.

Step 2: Build the cache key correctly

The cache key must capture everything that affects the output. A common mistake is keying only on the user's message and ignoring the system prompt, model version, or parameters like temperature — then getting cache hits that return stale or inconsistent answers.

A reliable approach is to hash the full request payload:

import { createHash } from "crypto";

function buildCacheKey(request) {
  const normalized = JSON.stringify({
    model: request.model,
    system: request.system,
    messages: request.messages,
    temperature: request.temperature ?? 0,
    max_tokens: request.max_tokens,
  });
  return createHash("sha256").update(normalized).digest("hex");
}

Set temperature: 0 (or close to it) for cacheable paths — caching high-temperature creative outputs defeats the purpose, since you want variation there.

Step 3: Pick a storage layer and TTL

For most apps, Redis is the right tool: fast reads, built-in expiration, and easy to run alongside your app.

async function getCachedOrFetch(key, fetchFn, ttlSeconds = 3600) {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const result = await fetchFn();
  await redis.set(key, JSON.stringify(result), "EX", ttlSeconds);
  return result;
}

TTL depends on how often the underlying answer should change:

• Static knowledge (product docs, policies): hours to days.
• Semi-dynamic content (summaries of recent data): minutes to an hour.
• User-specific or time-sensitive outputs: don't cache, or cache for seconds only.

Step 4: Decide between exact-match and semantic caching

Exact-match caching (hashing the literal prompt) is simple and safe but misses near-duplicate queries — "What's your refund policy?" and "How do refunds work?" won't match even though they'd produce nearly identical answers.

Semantic caching solves this by embedding the incoming query, searching a vector store for similar past queries above a similarity threshold, and returning the cached response if found. It catches more duplicates but adds complexity and risk: a too-loose similarity threshold returns wrong answers to different questions. If you go this route, start with a high threshold (0.92+) and tune down carefully while monitoring for mismatches.

For most teams, exact-match caching on normalized input (lowercased, whitespace-trimmed, stripped of irrelevant metadata) covers 80% of the savings with a fraction of the engineering effort. Add semantic caching later if duplicate-but-reworded traffic is actually a measured problem, not a guess.

Step 5: Invalidate deliberately

A cache that never expires correctly is worse than no cache. Build invalidation triggers for:

• Content updates (new docs, policy changes) — flush related keys or bump a cache-version prefix.
• Model upgrades — include the model identifier in the cache key so a model switch doesn't serve stale outputs from the old model.
• Manual overrides — give yourself an admin endpoint or CLI flag to purge specific keys when something's clearly wrong.

Combine caching with request-level cost controls

Caching reduces repeat-call costs, but you still need visibility into what you're spending on cache misses. If you're routing LLM calls through SubToAPI, every request already returns usage metadata (input/output tokens) alongside the response, so you can log cache-miss cost per route and see exactly where caching would pay off most. It also gives you a stable API surface — application keys, streaming, and tool use — so adding a caching layer in front doesn't mean restructuring how your app talks to the model. See the quickstart and messages docs for the request/response shape you'd be caching against.

A simple pattern: wrap your API call in the cache check, and only hit the live endpoint on a miss.

async function askModel(prompt) {
  const key = buildCacheKey({ model: "claude-sonnet", messages: [{ role: "user", content: prompt }] });
  return getCachedOrFetch(key, () =>
    fetch("https://api.subtoapi.app/v1/messages", {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "claude-sonnet",
        max_tokens: 500,
        messages: [{ role: "user", content: prompt }],
      }),
    }).then(r => r.json())
  );
}

Questions

Does caching hurt response quality? Not if scoped correctly. Cache only deterministic, low-personalization queries, and keep temperature low on cacheable paths so repeated calls would have produced the same answer anyway.

How much can caching actually save? It depends on query repetition. Support and FAQ-style apps with high duplicate traffic often see 30–80% cost reduction; highly unique, personalized workloads see much less.

Should I cache streaming responses? Yes — cache the fully assembled response after the stream completes, then replay it as a fast non-streamed (or simulated-stream) response on cache hits. See streaming docs for the response format to reconstruct.