Retry requests safely with idempotency
Keep one operation key across network retries to recover the same completed answer without another provider call or charge.
Choose one key per operation
Send Idempotency-Key on the first request and every retry. Generate a UUID once, before the retry loop, and keep it until the operation succeeds or is deliberately abandoned. A new task needs a new key. The header must occur once and contain 1 to 255 printable ASCII characters. Avoid personal data or secrets in the value.
Keys are scoped to one API key identity. Another API key cannot retrieve the result, including a second key in the same organization. The fingerprint covers the HTTP method, path and normalized request body. JSON object key order and whitespace do not matter. Changed values, array order or endpoint need a new key. Non-JSON bodies require identical decoded bytes and content type.
Send the request
Set TOKENLANE_API_KEY to the gateway key. Install openai with npm install openai or python -m pip install openai. The examples use a current model from the catalog. To repeat them across separate processes, set the same TOKENLANE_OPERATION_ID. Clear that variable before a new logical operation. The example disables automatic SDK retries so a second retry loop does not multiply attempts.
curl
curl --fail-with-body --request POST \
--url 'https://api.tokenlane.de/v1/chat/completions' \
--header "Authorization: Bearer $TOKENLANE_API_KEY" \
--header "Idempotency-Key: ${TOKENLANE_OPERATION_ID:-$(python3 -c 'import uuid; print(uuid.uuid4())')}" \
--header 'Content-Type: application/json' \
--data '{"model":"deepseek/deepseek-v4-flash-0731","messages":[{"role":"user","content":"Hello"}],"max_tokens":400}'
node
import OpenAI from 'openai';
import { randomUUID } from 'node:crypto';
const client = new OpenAI({
baseURL: "https://api.tokenlane.de/v1",
apiKey: process.env.TOKENLANE_API_KEY,
maxRetries: 0,
});
const key = process.env.TOKENLANE_OPERATION_ID ?? randomUUID();
const body = {
"model": "deepseek/deepseek-v4-flash-0731",
"messages": [
{
"role": "user",
"content": "Hello"
}
],
"max_tokens": 400
};
const { data: completion, response } = await client.chat.completions
.create(body, { headers: { 'Idempotency-Key': key } })
.withResponse();
console.log({ generation: completion.id, replayed: response.headers.get('Idempotent-Replayed') === 'true' });
python
import json
import os
import uuid
from openai import OpenAI
client = OpenAI(
base_url="https://api.tokenlane.de/v1",
api_key=os.environ["TOKENLANE_API_KEY"],
max_retries=0,
)
key = os.environ.get("TOKENLANE_OPERATION_ID") or str(uuid.uuid4())
body = json.loads("{\"model\":\"deepseek/deepseek-v4-flash-0731\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"max_tokens\":400}")
response = client.chat.completions.with_raw_response.create(
**body, extra_headers={"Idempotency-Key": key}
)
completion = response.parse()
print({"generation": completion.id, "replayed": response.headers.get("Idempotent-Replayed") == "true"})
vercel
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';
import { randomUUID } from 'node:crypto';
const key = process.env.TOKENLANE_OPERATION_ID ?? randomUUID();
const tokenlane = createOpenAICompatible({
name: 'tokenlane', baseURL: "https://api.tokenlane.de/v1",
apiKey: process.env.TOKENLANE_API_KEY, includeUsage: true,
headers: { 'Idempotency-Key': key },
});
const result = await generateText({
model: tokenlane("deepseek/deepseek-v4-flash-0731"),
messages: [{ role: 'user', content: 'Hello' }],
providerOptions: { tokenlane: { max_tokens: 400 } },
maxRetries: 0,
});
console.log({ generated: result.text.length > 0 });
A replay preserves the original status, body, generation ID, request ID and relevant response headers. Idempotent-Replayed: true identifies it. It creates no new generation and costs nothing. A request without Idempotency-Key keeps ordinary inference behavior and does not opt into this response store.
Handle the result
| Situation | Result | Next step |
|---|---|---|
| First request | One provider execution and its normal response | Keep the key for possible retries |
| Same completed request | Identical response and Idempotent-Replayed: true | Use the original generation ID |
| Different content with the same key | 409 idempotency_conflict | Use a new key for the changed operation |
| Original still running | 409 idempotency_in_progress and Retry-After | Wait, then retry the identical request |
| Retention expired | A new, billable execution is possible | Avoid retries outside the retention window |
| Response too large or safe replay unavailable | 409 idempotency_unavailable on repetition | Do not retry automatically with a new key |
| Replay encryption is not configured | 501 feature_unavailable, retryable: false, no Retry-After | Stop; ask the operator to configure replay before trying again |
| Retryable HTTP 5xx, or a confirmed client abort | No completed result is stored | Retry the same key with bounded backoff; never override metadata.retryable: false |
| HTTP 4xx | The error is stored and repeated | Correct the cause and use a new operation key |
The default response limit is 8 MiB. An oversized first answer is still delivered, but cannot be recovered from this store. Local memory or shared-capacity pressure can also prevent replay. Follow remedy_hint; a new key deliberately starts a new operation and may be billed. A cached 429 remains the same cached error: Retry-After does not remove it.
Authentication, invalid header and transport/body-limit checks happen before a usable fingerprint and are not replay records. During a shared-state outage, an unfinished original may remain in progress until its safety lease expires. A lost cache is never interpreted as permission to immediately repeat paid work.
Bound retries with backoff
The pattern below retries connection failures, retryable server errors and an operation still in progress. An explicit metadata.retryable: false stops the loop even for HTTP 5xx, including 501 feature_unavailable. Other 409 and 4xx responses also stop the loop. It keeps one key, respects Retry-After, caps the number of attempts and stops rather than shortening a server-requested pause beyond the deadline. Set TOKENLANE_MODEL to a current model ID from the catalog.
import os
import random
import time
import uuid
from openai import OpenAI, APIConnectionError, APIStatusError
client = OpenAI(
base_url=os.environ.get("TOKENLANE_BASE_URL", "https://api.tokenlane.de/v1"),
api_key=os.environ["TOKENLANE_API_KEY"],
max_retries=0,
)
key = str(uuid.uuid4())
body = {
"model": os.environ["TOKENLANE_MODEL"],
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 400,
}
deadline = time.monotonic() + 45
for attempt in range(5):
remaining = deadline - time.monotonic()
if remaining <= 0:
raise TimeoutError("Retry deadline reached")
retry_after = 0.0
try:
result = client.chat.completions.create(
**body,
extra_headers={"Idempotency-Key": key},
timeout=min(15.0, remaining),
)
print(result.id)
break
except APIConnectionError:
if attempt == 4:
raise
except APIStatusError as error:
detail = error.body if isinstance(error.body, dict) else {}
detail = detail.get("error", detail)
detail = detail if isinstance(detail, dict) else {}
metadata = detail.get("metadata")
metadata = metadata if isinstance(metadata, dict) else {}
kind = metadata.get("type")
transient = metadata.get("retryable") is not False and (
error.status_code >= 500
or (error.status_code == 409 and kind == "idempotency_in_progress")
)
if not transient or attempt == 4:
raise
value = error.response.headers.get("Retry-After", "0")
if not value.isascii() or not value.isdecimal():
raise
retry_after = float(value)
delay = max(retry_after, min(8.0, 0.5 * 2**attempt) + random.uniform(0, 0.25))
if delay >= deadline - time.monotonic():
raise TimeoutError("Retry-After exceeds retry deadline")
time.sleep(delay)
Complete a stream before replaying it
The first stream byte is forwarded immediately. Only a fully completed successful stream can be replayed, including usage events and [DONE]. An interrupted or failed stream is not a completed result. A replay starts from the beginning; it does not resume after the last received token. Discard partial text before replacing it with a recovered complete answer. Never silently append a new stream to partial output.
Understand storage and privacy
The header explicitly opts this request into storing the full response. It is encrypted in Valkey for at most ten minutes after completion; the default is ten minutes and operators may shorten it. It is not written to the GenerationStore by idempotency, does not grant capture permission and is not exposed through GET /v1/generation/content. Generation capture remains a separate opt-in. No prompt, answer, authorization header or idempotency header value belongs in logs or metrics.
Keep the key and request stable and retry promptly. After expiry, the same key may start another billable operation. See errors and retries, privacy and compliance and the error catalog.
SDK installation and client setup: official OpenAI SDK documentation. The replay and billing behavior described here is the Tokenlane gateway contract.