Cache | Alephant Documentation

Alephant AI Gateway can reuse eligible model responses so repeated requests avoid unnecessary upstream calls. This can reduce latency and provider cost for stable, repeatable workloads.

Cache controls are optional gateway headers. The gateway consumes them during request processing and does not send them to the upstream model provider. Your application is still responsible for deciding whether a response is safe to reuse.

Cache modes

Alephant uses the word cache for several different mechanisms. They do not have the same behavior or lifecycle.

Mode	Purpose
Exact response cache	Reuses a response when a request produces the same cache identity.
Semantic cache	Looks for a sufficiently similar request when semantic caching and embedding support are configured for the workspace.
Provider-native prompt caching	Reuses prompt tokens inside an upstream provider. Provider usage and pricing report this behavior; it is not an Alephant Gateway response-cache hit.
Internal operational state	Supports routing, policy context, idempotency, sessions, and gateway protection. It is not a reusable model response.

When to use caching

Good candidates have stable inputs and responses that remain valid for more than one request:

Knowledge answers based on content that changes infrequently
Repeated classification, extraction, or summarization
Fixed prompt templates with repeatable variables
Read-heavy workloads where lower latency matters

Use caching carefully when:

The answer must reflect real-time data
The request contains sensitive or user-specific information
The request can invoke tools or create external side effects
Authorization or business state can change between requests
The application depends on deliberately random output

The gateway cannot determine every business-specific reuse boundary. Disable cache reads and writes when a response must be generated from the current request.

Configure response caching

Core cache headers

Header	Usage
`Alephant-Cache-Enabled`	Enables cache behavior when set to `true`.
`Alephant-Cache-Read`	Allows reading from the cache when set to `true`.
`Alephant-Cache-Save`	Allows saving eligible responses when set to `true`.
`Alephant-Cache-Bucket-Max-Size`	Optional bucket count. The valid range is `1-20`.
`Alephant-Cache-Seed`	Optional seed that participates in cache identity derivation. Use it to separate known application contexts.
`Alephant-Cache-Control`	Optional cache control, such as `max-age` or `s-maxage`.

Read and save behavior

Set read and save controls explicitly when the application needs predictable behavior.

Enabled	Read	Save	Behavior
`true`	`true`	`true`	Try the cache first; an eligible miss may be saved.
`true`	`true`	`false`	Read an existing response without saving a miss.
`true`	`false`	`true`	Skip lookup and allow an eligible response to be written.
Not enabled	Any	Any	Do not use the other controls to activate response caching.

Workspace and gateway configuration can affect cache availability. Do not rely on undocumented defaults when a particular read or save behavior matters.

Semantic cache headers

Semantic caching is an advanced control. Use it only when the workspace has semantic caching and an embedding service configured.

Header	Usage
`Alephant-Embeddings-Model`	Embedding model used for semantic-cache lookup.
`Alephant-Embeddings-Key`	Embedding credential used by the gateway for the lookup.
`Alephant-Cache-Semantic-Threshold`	Similarity threshold for semantic matching.
`Alephant-Cache-Ttl`	TTL used by semantic caching.

Treat the embedding credential as a secret. Do not commit it to source control or include it in application logs.

Example

The following request explicitly enables cache reads and writes and gives the cached response a one-hour maximum age:

$ curl "$ALEPHANT_GATEWAY_BASE_URL/v1/chat/completions" \
>   -H "Authorization: Bearer $ALEPHANT_VIRTUAL_KEY" \
>   -H "Content-Type: application/json" \
>   -H "Alephant-Cache-Enabled: true" \
>   -H "Alephant-Cache-Read: true" \
>   -H "Alephant-Cache-Save: true" \
>   -H "Alephant-Cache-Control: max-age=3600" \
>   -d '{
>     "model": "openai/gpt-4o-mini",
>     "messages": [
>       {
>         "role": "user",
>         "content": "Summarize the refund policy in three bullets."
>       }
>     ]
>   }'

Send the same stable input again, then compare the cache response metadata and the request in Logs. The max-age=3600 value is explicit for this example; it does not describe an implicit gateway lifetime.

Request lifecycle

For a cache-enabled request, the logical flow is:

Authenticate the Virtual Key and resolve workspace, model, and request context.
Derive a cache identity from the applicable request and cache controls.
If reads are allowed, attempt an exact or configured semantic lookup.
Return the reusable response with cache metadata when the lookup succeeds.
On a miss, send the request to the selected upstream provider.
If saving is allowed and the response is eligible, write it to the response cache.
Continue recording the request in the configured Logs, Analytics, and cost-attribution paths.

These are logical stages, not a promise about internal thread order, Redis operations, or strong consistency.

Verify a cache hit

Cache-enabled responses may include:

Response header	Usage
`alephant-cache`	Cache status for the request.
`alephant-cache-bucket-idx`	Cache bucket index for a returned cache entry.
`alephant-cache-latency`	Cache lookup latency.
`alephant-id`	Alephant request identifier, especially useful for a response returned from cache.

To verify behavior:

Send the same stable request twice with cache reads and writes enabled.
Compare alephant-cache, cache latency, and the Alephant request identifier.
Open Logs and inspect cache status, provider, tokens, cost, and request context.
If the second request misses, compare the model, messages, parameters, seed, TTL, and cache controls.

The exact values reported by alephant-cache can depend on the deployed gateway version. Inspect the actual response instead of depending on an undocumented enum.

Troubleshooting

Symptom	What to check
Identical-looking prompts still miss	Compare the full request, model, parameters, seed, TTL, workspace, and cache controls.
A request hits when saving is disabled	`Alephant-Cache-Save: false` does not prevent `Alephant-Cache-Read: true` from reading an existing entry.
Semantic lookup does not run	Confirm that the workspace, embedding model, embedding credential, and similarity threshold are configured.
A response could cross a business boundary	Disable reads and saves, or use an explicit seed to isolate a well-defined application context.
Cache behavior differs between deployments	Compare gateway version, workspace configuration, and the backing cache service.

Internal cache and state

The gateway also stores short-lived or reusable operational state. These mechanisms support request processing but are separate from the model response cache.

Category	Internal state	User-visible purpose
Request and policy context	Prompt Cache, PII Cache flag, VK Enrichment Cache	Resolve prompt templates and enrich policy or log context with workspace and department metadata.
Model and semantic discovery	Model Catalog Cache, Embedding Base URL Cache	Accelerate model-support lookups and select the configured embedding service.
Async task consistency	Image Task Affinity, Image Task Cost Claim	Reuse the provider path selected when an image task was created and prevent duplicate cost claims.
Agent and tool sessions	MCP Session Cache, MCP Session Lock, Agent Step State	Reuse MCP sessions, coordinate initialization, and detect repeated or conflicting steps.
Endpoint and regional routing	x402 Endpoint Snapshot, x402 Signing Secret, Regional Endpoint	Resolve endpoint configuration, protected signing material, and an established regional route.
Gateway protection	Workspace Concurrency, Client IP Rate Limit, Gateway In-flight Limit	Enforce workspace, client-IP, and gateway-wide traffic protection.

These states do not share one TTL or invalidation rule. A miss may trigger a database lookup or reinitialization and is not necessarily an error. Concurrency, rate-limit, lock, and idempotency state never represents a reusable model response. Their keys, values, and lifecycles are deployment details rather than public API contracts.

Cache, Logs, and billing

Logs and accounting paths maintain other caches and aggregates for model prices, custom prices, workspace or department spend, Virtual Key spend, token and request counts, Agent cost, usage-limit decisions, authentication, and common query results.

Keep these events separate when investigating cost:

A provider-native cached-token event is reported through provider usage and pricing.
A Gateway response-cache hit reuses a response before another upstream generation.
An internal price-cache hit only accelerates cost calculation.
Rate-limit and spend aggregates support enforcement and reporting; they are not response caches.

A cache hit should still have observable metadata. Use the actual Logs and Analytics records to determine provider tokens, gateway charges, cache savings, and attributed cost for a request.

Security and operational guidance

Do not cache authorization decisions, one-time results, or sensitive responses that must not be reused across requests.
Do not put secrets, personal data, or regulated identifiers in a cache seed or custom observable metadata.
Applications must use the documented gateway request and response headers instead of depending on internal Redis keys.
In private deployments, protect Redis or Valkey as infrastructure: restrict network access, require authentication, encrypt transport where supported, and avoid exposing operational values in logs.
When changing cache configuration, verify behavior with non-sensitive test requests before enabling it broadly.