Retry requests safely with idempotency

Keep one operation key across network retries to recover the same completed answer without another provider call or charge.

Choose one key per operation

Send Idempotency-Key on the first request and every retry. Generate a UUID once, before the retry loop, and keep it until the operation succeeds or is deliberately abandoned. A new task needs a new key. The header must occur once and contain 1 to 255 printable ASCII characters. Avoid personal data or secrets in the value.

Keys are scoped to one API key identity. Another API key cannot retrieve the result, including a second key in the same organization. The fingerprint covers the HTTP method, path and normalized request body. JSON object key order and whitespace do not matter. Changed values, array order or endpoint need a new key. Non-JSON bodies require identical decoded bytes and content type.

Send the request

Set TOKENLANE_API_KEY to the gateway key. Install openai with npm install openai or python -m pip install openai. The examples use a current model from the catalog. To repeat them across separate processes, set the same TOKENLANE_OPERATION_ID. Clear that variable before a new logical operation. The example disables automatic SDK retries so a second retry loop does not multiply attempts.

curl

curl --fail-with-body --request POST \
  --url 'https://api.tokenlane.de/v1/chat/completions' \
  --header "Authorization: Bearer $TOKENLANE_API_KEY" \
  --header "Idempotency-Key: ${TOKENLANE_OPERATION_ID:-$(python3 -c 'import uuid; print(uuid.uuid4())')}" \
  --header 'Content-Type: application/json' \
  --data '{"model":"deepseek/deepseek-v4-flash-0731","messages":[{"role":"user","content":"Hello"}],"max_tokens":400}'

node

import OpenAI from 'openai';
import { randomUUID } from 'node:crypto';

const client = new OpenAI({
  baseURL: "https://api.tokenlane.de/v1",
  apiKey: process.env.TOKENLANE_API_KEY,
  maxRetries: 0,
});
const key = process.env.TOKENLANE_OPERATION_ID ?? randomUUID();
const body = {
  "model": "deepseek/deepseek-v4-flash-0731",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "max_tokens": 400
};
const { data: completion, response } = await client.chat.completions
  .create(body, { headers: { 'Idempotency-Key': key } })
  .withResponse();
console.log({ generation: completion.id, replayed: response.headers.get('Idempotent-Replayed') === 'true' });

python

import json
import os
import uuid
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokenlane.de/v1",
    api_key=os.environ["TOKENLANE_API_KEY"],
    max_retries=0,
)
key = os.environ.get("TOKENLANE_OPERATION_ID") or str(uuid.uuid4())
body = json.loads("{\"model\":\"deepseek/deepseek-v4-flash-0731\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"max_tokens\":400}")
response = client.chat.completions.with_raw_response.create(
    **body, extra_headers={"Idempotency-Key": key}
)
completion = response.parse()
print({"generation": completion.id, "replayed": response.headers.get("Idempotent-Replayed") == "true"})

vercel

import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
import { generateText } from 'ai';
import { randomUUID } from 'node:crypto';

const key = process.env.TOKENLANE_OPERATION_ID ?? randomUUID();
const tokenlane = createOpenAICompatible({
  name: 'tokenlane', baseURL: "https://api.tokenlane.de/v1",
  apiKey: process.env.TOKENLANE_API_KEY, includeUsage: true,
  headers: { 'Idempotency-Key': key },
});
const result = await generateText({
  model: tokenlane("deepseek/deepseek-v4-flash-0731"),
  messages: [{ role: 'user', content: 'Hello' }],
  providerOptions: { tokenlane: { max_tokens: 400 } },
  maxRetries: 0,
});
console.log({ generated: result.text.length > 0 });

A replay preserves the original status, body, generation ID, request ID and relevant response headers. Idempotent-Replayed: true identifies it. It creates no new generation and costs nothing. A request without Idempotency-Key keeps ordinary inference behavior and does not opt into this response store.

Handle the result

SituationResultNext step
First requestOne provider execution and its normal responseKeep the key for possible retries
Same completed requestIdentical response and Idempotent-Replayed: trueUse the original generation ID
Different content with the same key409 idempotency_conflictUse a new key for the changed operation
Original still running409 idempotency_in_progress and Retry-AfterWait, then retry the identical request
Retention expiredA new, billable execution is possibleAvoid retries outside the retention window
Response too large or safe replay unavailable409 idempotency_unavailable on repetitionDo not retry automatically with a new key
Replay encryption is not configured501 feature_unavailable, retryable: false, no Retry-AfterStop; ask the operator to configure replay before trying again
Retryable HTTP 5xx, or a confirmed client abortNo completed result is storedRetry the same key with bounded backoff; never override metadata.retryable: false
HTTP 4xxThe error is stored and repeatedCorrect the cause and use a new operation key

The default response limit is 8 MiB. An oversized first answer is still delivered, but cannot be recovered from this store. Local memory or shared-capacity pressure can also prevent replay. Follow remedy_hint; a new key deliberately starts a new operation and may be billed. A cached 429 remains the same cached error: Retry-After does not remove it.

Authentication, invalid header and transport/body-limit checks happen before a usable fingerprint and are not replay records. During a shared-state outage, an unfinished original may remain in progress until its safety lease expires. A lost cache is never interpreted as permission to immediately repeat paid work.

Bound retries with backoff

The pattern below retries connection failures, retryable server errors and an operation still in progress. An explicit metadata.retryable: false stops the loop even for HTTP 5xx, including 501 feature_unavailable. Other 409 and 4xx responses also stop the loop. It keeps one key, respects Retry-After, caps the number of attempts and stops rather than shortening a server-requested pause beyond the deadline. Set TOKENLANE_MODEL to a current model ID from the catalog.

import os
import random
import time
import uuid
from openai import OpenAI, APIConnectionError, APIStatusError

client = OpenAI(
    base_url=os.environ.get("TOKENLANE_BASE_URL", "https://api.tokenlane.de/v1"),
    api_key=os.environ["TOKENLANE_API_KEY"],
    max_retries=0,
)
key = str(uuid.uuid4())
body = {
    "model": os.environ["TOKENLANE_MODEL"],
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 400,
}
deadline = time.monotonic() + 45
for attempt in range(5):
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        raise TimeoutError("Retry deadline reached")
    retry_after = 0.0
    try:
        result = client.chat.completions.create(
            **body,
            extra_headers={"Idempotency-Key": key},
            timeout=min(15.0, remaining),
        )
        print(result.id)
        break
    except APIConnectionError:
        if attempt == 4:
            raise
    except APIStatusError as error:
        detail = error.body if isinstance(error.body, dict) else {}
        detail = detail.get("error", detail)
        detail = detail if isinstance(detail, dict) else {}
        metadata = detail.get("metadata")
        metadata = metadata if isinstance(metadata, dict) else {}
        kind = metadata.get("type")
        transient = metadata.get("retryable") is not False and (
            error.status_code >= 500
            or (error.status_code == 409 and kind == "idempotency_in_progress")
        )
        if not transient or attempt == 4:
            raise
        value = error.response.headers.get("Retry-After", "0")
        if not value.isascii() or not value.isdecimal():
            raise
        retry_after = float(value)
    delay = max(retry_after, min(8.0, 0.5 * 2**attempt) + random.uniform(0, 0.25))
    if delay >= deadline - time.monotonic():
        raise TimeoutError("Retry-After exceeds retry deadline")
    time.sleep(delay)

Complete a stream before replaying it

The first stream byte is forwarded immediately. Only a fully completed successful stream can be replayed, including usage events and [DONE]. An interrupted or failed stream is not a completed result. A replay starts from the beginning; it does not resume after the last received token. Discard partial text before replacing it with a recovered complete answer. Never silently append a new stream to partial output.

Understand storage and privacy

The header explicitly opts this request into storing the full response. It is encrypted in Valkey for at most ten minutes after completion; the default is ten minutes and operators may shorten it. It is not written to the GenerationStore by idempotency, does not grant capture permission and is not exposed through GET /v1/generation/content. Generation capture remains a separate opt-in. No prompt, answer, authorization header or idempotency header value belongs in logs or metrics.

Keep the key and request stable and retry promptly. After expiry, the same key may start another billable operation. See errors and retries, privacy and compliance and the error catalog.

SDK installation and client setup: official OpenAI SDK documentation. The replay and billing behavior described here is the Tokenlane gateway contract.