Reduce LLM API Costs With Caching: A Practical Guide

If you're running an LLM-powered product, your API bill is probably growing faster than your user base. The fastest way to reduce LLM API costs without touching your UX is caching: storing and reusing responses (or parts of requests) so you don't pay to regenerate the same tokens twice.

There are three distinct caching strategies that apply to LLM APIs, and most cost-conscious teams use all three together: prompt caching (reusing large static context across calls), response caching (storing full outputs for repeated inputs), and semantic caching (matching requests that are similar, not identical). Each one targets a different source of waste in your token spend. Below is a practical breakdown of when to use each, with working examples.

Why LLM costs balloon without caching

Every request to a model API charges you for input tokens and output tokens. If your app sends a long system prompt, a set of tool definitions, or a document as context on every single call, you're paying for those same tokens repeatedly — even when nothing in them changed. A support bot that includes a 3,000-token knowledge base snippet in every request is burning most of its budget on context the model has already "seen" a thousand times that day.

The fix isn't to shrink your prompts (that hurts quality). It's to stop paying full price for tokens that don't change.

Prompt caching: reuse static context cheaply

Prompt caching lets you mark a portion of your input — system instructions, tool schemas, reference documents — as reusable. On subsequent calls within a short time window, the provider charges a reduced rate for that cached portion instead of the full input rate.

This is the single biggest lever for apps with large, mostly-static context:

const response = await fetch("https://api.subtoapi.app/v1/messages", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SUBTOAPI_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    model: "claude-sonnet-4",
    system: longSystemPromptWithDocs,
    messages: [{ role: "user", content: userQuestion }],
    max_tokens: 1024
  })
});

The practical win here: if your system prompt or tool definitions rarely change between requests, structure your calls so that static content comes first and user-specific content comes last. This maximizes what can be reused and minimizes what gets billed at full rate. Check your provider's docs — see /docs/messages for request structure — for exact cache lifetime and eligibility rules.

Response caching: don't regenerate identical answers

Response caching is simpler: if the exact same input has already produced an output, serve the stored output instead of calling the model again. This works well for:

• FAQ-style endpoints where many users ask the same question
• Documentation search assistants
• Any feature where identical inputs are common (autocomplete, classification, tagging)

A minimal implementation using a key-value store:

import crypto from "crypto";

function cacheKey(prompt, model) {
  return crypto.createHash("sha256").update(model + prompt).digest("hex");
}

async function getCachedOrGenerate(prompt, model, kv) {
  const key = cacheKey(prompt, model);
  const cached = await kv.get(key);
  if (cached) return JSON.parse(cached);

  const result = await callLLM(prompt, model);
  await kv.set(key, JSON.stringify(result), { ex: 86400 }); // 24h TTL
  return result;
}

This is exact-match caching: a single character difference in the prompt produces a cache miss. It's cheap to implement and gives you free wins on high-repetition workloads, but it won't help if your inputs vary even slightly — which is most real-world traffic.

Semantic caching: catch near-duplicate requests

Semantic caching solves the "slightly different wording, same intent" problem. Instead of hashing the raw prompt, you embed it into a vector, compare it against previously cached embeddings, and reuse the stored response if similarity is above a threshold.

async function semanticCacheLookup(prompt, vectorStore, threshold = 0.92) {
  const embedding = await embed(prompt);
  const match = await vectorStore.query(embedding, { topK: 1 });

  if (match && match.score >= threshold) {
    return match.metadata.response;
  }
  return null;
}

This adds infrastructure (an embedding model and a vector store) and some risk — two questions can be semantically close but require different answers, so you need a conservative threshold and good test coverage before relying on it in production. It's worth the complexity for high-volume consumer-facing chat products where the same handful of questions get rephrased thousands of ways.

Pick the right layer for your workload

A rough decision guide:

| Workload pattern | Best caching strategy | |---|---| | Large static system prompt / tool definitions | Prompt caching | | Exact repeated queries (FAQ, lookups) | Response caching | | Rephrased but semantically similar queries | Semantic caching | | Streaming, highly personalized responses | None of the above — cache only shared context |

Most production systems stack these: prompt caching on the system layer, response caching on the application layer, and semantic caching on the user-facing layer for high-traffic features.

Caching is one lever — infrastructure is another

Caching reduces how much you pay per request. The other side of cost control is not overpaying for infrastructure around those requests — retries, logging, routing, and multi-provider fallback logic that teams often build and maintain themselves. If you're consolidating access to Claude across a team, SubToAPI turns your existing Claude access into a standard HTTPS API with per-app keys (sub_live_...), streaming, tool use, and usage metadata per key — so you can see exactly which app or feature is driving token spend before you decide what to cache. Plans start at €9/month for solo use, with team seats on the Team (€19/seat) and Scale (€49/seat) plans — see /pricing. There's a free trial at /signup.

Start with the quickstart if you're wiring up a new integration, and check /docs/streaming if your caching layer needs to work alongside streamed responses — cached hits should still be delivered in a format your frontend expects.

questions

Does caching reduce output token costs too? Prompt caching only discounts input tokens you've marked as reusable. Output tokens are generated fresh each time unless the entire response is served from a response or semantic cache, which skips the model call entirely.

Is semantic caching safe for factual or compliance-sensitive answers? Use a high similarity threshold and validate matches before serving them, or avoid semantic caching for that traffic. Two similar-sounding questions can require materially different answers, so exact or prompt caching is safer when accuracy matters more than cost.

How much can caching actually save? It depends on how repetitive your traffic is. Apps with large static prompts and high context reuse often cut input costs significantly; apps with highly personalized, one-off prompts will see smaller gains. Measure your own hit rate before assuming a specific savings number.