> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://developers.alephant.io/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://developers.alephant.io/_mcp/server.

# Cache

> Reuse safe model responses to reduce latency and upstream cost

Alephant AI Gateway can reuse eligible model responses so repeated requests avoid unnecessary upstream calls. This can reduce latency and provider cost for stable, repeatable workloads.

Cache controls are optional gateway headers. The gateway consumes them during request processing and does not send them to the upstream model provider. Your application is still responsible for deciding whether a response is safe to reuse.

## Cache modes

Alephant uses the word *cache* for several different mechanisms. They do not have the same behavior or lifecycle.

| Mode                           | Purpose                                                                                                                                              |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Exact response cache           | Reuses a response when a request produces the same cache identity.                                                                                   |
| Semantic cache                 | Looks for a sufficiently similar request when semantic caching and embedding support are configured for the workspace.                               |
| Provider-native prompt caching | Reuses prompt tokens inside an upstream provider. Provider usage and pricing report this behavior; it is not an Alephant Gateway response-cache hit. |
| Internal operational state     | Supports routing, policy context, idempotency, sessions, and gateway protection. It is not a reusable model response.                                |

## When to use caching

Good candidates have stable inputs and responses that remain valid for more than one request:

* Knowledge answers based on content that changes infrequently
* Repeated classification, extraction, or summarization
* Fixed prompt templates with repeatable variables
* Read-heavy workloads where lower latency matters

Use caching carefully when:

* The answer must reflect real-time data
* The request contains sensitive or user-specific information
* The request can invoke tools or create external side effects
* Authorization or business state can change between requests
* The application depends on deliberately random output

The gateway cannot determine every business-specific reuse boundary. Disable cache reads and writes when a response must be generated from the current request.

## Configure response caching

### Core cache headers

| Header                           | Usage                                                                                                        |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `Alephant-Cache-Enabled`         | Enables cache behavior when set to `true`.                                                                   |
| `Alephant-Cache-Read`            | Allows reading from the cache when set to `true`.                                                            |
| `Alephant-Cache-Save`            | Allows saving eligible responses when set to `true`.                                                         |
| `Alephant-Cache-Bucket-Max-Size` | Optional bucket count. The valid range is `1-20`.                                                            |
| `Alephant-Cache-Seed`            | Optional seed that participates in cache identity derivation. Use it to separate known application contexts. |
| `Alephant-Cache-Control`         | Optional cache control, such as `max-age` or `s-maxage`.                                                     |

### Read and save behavior

Set read and save controls explicitly when the application needs predictable behavior.

| Enabled     | Read    | Save    | Behavior                                                    |
| ----------- | ------- | ------- | ----------------------------------------------------------- |
| `true`      | `true`  | `true`  | Try the cache first; an eligible miss may be saved.         |
| `true`      | `true`  | `false` | Read an existing response without saving a miss.            |
| `true`      | `false` | `true`  | Skip lookup and allow an eligible response to be written.   |
| Not enabled | Any     | Any     | Do not use the other controls to activate response caching. |

Workspace and gateway configuration can affect cache availability. Do not rely on undocumented defaults when a particular read or save behavior matters.

### Semantic cache headers

Semantic caching is an advanced control. Use it only when the workspace has semantic caching and an embedding service configured.

| Header                              | Usage                                                    |
| ----------------------------------- | -------------------------------------------------------- |
| `Alephant-Embeddings-Model`         | Embedding model used for semantic-cache lookup.          |
| `Alephant-Embeddings-Key`           | Embedding credential used by the gateway for the lookup. |
| `Alephant-Cache-Semantic-Threshold` | Similarity threshold for semantic matching.              |
| `Alephant-Cache-Ttl`                | TTL used by semantic caching.                            |

Treat the embedding credential as a secret. Do not commit it to source control or include it in application logs.

## Example

The following request explicitly enables cache reads and writes and gives the cached response a one-hour maximum age:

```bash
curl "$ALEPHANT_GATEWAY_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $ALEPHANT_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -H "Alephant-Cache-Enabled: true" \
  -H "Alephant-Cache-Read: true" \
  -H "Alephant-Cache-Save: true" \
  -H "Alephant-Cache-Control: max-age=3600" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Summarize the refund policy in three bullets."
      }
    ]
  }'
```

Send the same stable input again, then compare the cache response metadata and the request in **Logs**. The `max-age=3600` value is explicit for this example; it does not describe an implicit gateway lifetime.

## Request lifecycle

For a cache-enabled request, the logical flow is:

1. Authenticate the Virtual Key and resolve workspace, model, and request context.
2. Derive a cache identity from the applicable request and cache controls.
3. If reads are allowed, attempt an exact or configured semantic lookup.
4. Return the reusable response with cache metadata when the lookup succeeds.
5. On a miss, send the request to the selected upstream provider.
6. If saving is allowed and the response is eligible, write it to the response cache.
7. Continue recording the request in the configured Logs, Analytics, and cost-attribution paths.

These are logical stages, not a promise about internal thread order, Redis operations, or strong consistency.

## Verify a cache hit

Cache-enabled responses may include:

| Response header             | Usage                                                                              |
| --------------------------- | ---------------------------------------------------------------------------------- |
| `alephant-cache`            | Cache status for the request.                                                      |
| `alephant-cache-bucket-idx` | Cache bucket index for a returned cache entry.                                     |
| `alephant-cache-latency`    | Cache lookup latency.                                                              |
| `alephant-id`               | Alephant request identifier, especially useful for a response returned from cache. |

To verify behavior:

1. Send the same stable request twice with cache reads and writes enabled.
2. Compare `alephant-cache`, cache latency, and the Alephant request identifier.
3. Open **Logs** and inspect cache status, provider, tokens, cost, and request context.
4. If the second request misses, compare the model, messages, parameters, seed, TTL, and cache controls.

The exact values reported by `alephant-cache` can depend on the deployed gateway version. Inspect the actual response instead of depending on an undocumented enum.

## Troubleshooting

| Symptom                                    | What to check                                                                                               |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| Identical-looking prompts still miss       | Compare the full request, model, parameters, seed, TTL, workspace, and cache controls.                      |
| A request hits when saving is disabled     | `Alephant-Cache-Save: false` does not prevent `Alephant-Cache-Read: true` from reading an existing entry.   |
| Semantic lookup does not run               | Confirm that the workspace, embedding model, embedding credential, and similarity threshold are configured. |
| A response could cross a business boundary | Disable reads and saves, or use an explicit seed to isolate a well-defined application context.             |
| Cache behavior differs between deployments | Compare gateway version, workspace configuration, and the backing cache service.                            |

## Internal cache and state

The gateway also stores short-lived or reusable operational state. These mechanisms support request processing but are separate from the model response cache.

| Category                      | Internal state                                                       | User-visible purpose                                                                               |
| ----------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Request and policy context    | Prompt Cache, PII Cache flag, VK Enrichment Cache                    | Resolve prompt templates and enrich policy or log context with workspace and department metadata.  |
| Model and semantic discovery  | Model Catalog Cache, Embedding Base URL Cache                        | Accelerate model-support lookups and select the configured embedding service.                      |
| Async task consistency        | Image Task Affinity, Image Task Cost Claim                           | Reuse the provider path selected when an image task was created and prevent duplicate cost claims. |
| Agent and tool sessions       | MCP Session Cache, MCP Session Lock, Agent Step State                | Reuse MCP sessions, coordinate initialization, and detect repeated or conflicting steps.           |
| Endpoint and regional routing | x402 Endpoint Snapshot, x402 Signing Secret, Regional Endpoint       | Resolve endpoint configuration, protected signing material, and an established regional route.     |
| Gateway protection            | Workspace Concurrency, Client IP Rate Limit, Gateway In-flight Limit | Enforce workspace, client-IP, and gateway-wide traffic protection.                                 |

These states do not share one TTL or invalidation rule. A miss may trigger a database lookup or reinitialization and is not necessarily an error. Concurrency, rate-limit, lock, and idempotency state never represents a reusable model response. Their keys, values, and lifecycles are deployment details rather than public API contracts.

## Cache, Logs, and billing

Logs and accounting paths maintain other caches and aggregates for model prices, custom prices, workspace or department spend, Virtual Key spend, token and request counts, Agent cost, usage-limit decisions, authentication, and common query results.

Keep these events separate when investigating cost:

* A provider-native cached-token event is reported through provider usage and pricing.
* A Gateway response-cache hit reuses a response before another upstream generation.
* An internal price-cache hit only accelerates cost calculation.
* Rate-limit and spend aggregates support enforcement and reporting; they are not response caches.

A cache hit should still have observable metadata. Use the actual **Logs** and **Analytics** records to determine provider tokens, gateway charges, cache savings, and attributed cost for a request.

## Security and operational guidance

* Do not cache authorization decisions, one-time results, or sensitive responses that must not be reused across requests.
* Do not put secrets, personal data, or regulated identifiers in a cache seed or custom observable metadata.
* Applications must use the documented gateway request and response headers instead of depending on internal Redis keys.
* In private deployments, protect Redis or Valkey as infrastructure: restrict network access, require authentication, encrypt transport where supported, and avoid exposing operational values in logs.
* When changing cache configuration, verify behavior with non-sensitive test requests before enabling it broadly.

## Related pages

* [Gateway Integration](/docs/overview/core-concepts/gateway-integration)
* [Prompt Management](/ai-gateway/prompt-manage)
* [Routing Optimization](/ai-gateway/routing-optimization)
* [Session](/ai-gateway/session)
* [Logs](/docs/overview/security-compliance/audit-logs)
* [Analytics](/docs/overview/fin-ops-budget/cost-analytics)