Prompt Caching in the API

Functional Positioning

Prompt Caching is a serving-layer optimization feature introduced by OpenAI in its API (v1.35+), designed for high-frequency, low-variance prompt usage—e.g., fixed system prompts, standardized RAG query templates, or batched instruction-based jobs.

Technical Implementation & Billing Logic

• Caching relies on semantic hashing (not exact string matching), tolerating minor perturbations (e.g., whitespace, punctuation, synonym substitution) while still yielding cache hits;
• On cache hit, input tokens are billed at 10% of standard rate (i.e., 90% discount), reflected explicitly in response fields usage.cache_creation_input_tokens and usage.cache_read_input_tokens;
• Default TTL is 24 hours; users may override it via the cache_control request parameter with ttl;
• Cache status is unambiguously signaled via the response header x-cache: values include HIT, MISS, and STALE.

Constraints & Limitations

• Supported only on gpt-4-turbo (gpt-4-turbo-2024-04-09 and later) and gpt-3.5-turbo (gpt-3.5-turbo-0125 and later);
• Cache reads are disabled under streaming mode;
• Caches are isolated across model versions and OpenAI organization boundaries;
• Requests containing "cache_control": {"type": "ephemeral"} bypass caching entirely.