Rerank
Order candidate documents by relevance to a query: request format, limits, costs and errors.
When to use it
Reranking is the second step of retrieval. A vector or keyword search returns, say, 100 candidates quickly; a rerank model reads the query together with each candidate and orders them much more precisely. Then you pass the best few to the chat model. Rerank does not search your database and does not generate an answer. For the first step see Embeddings.
Minimal example
A query and two documents; top_n limits the result to the best hit.
curl
curl --request POST \
--url 'https://api.tokenlane.de/v1/rerank' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "baai/bge-reranker-v2-m3",
"query": "European capital",
"documents": [
"Paris is in France.",
"A recipe for soup."
],
"top_n": 1
}'
node
const response = await fetch("https://api.tokenlane.de/v1/rerank", {
method: "POST",
headers: {
"Authorization": "Bearer " + process.env.TOKENLANE_API_KEY,
"Content-Type": "application/json",
},
body: `{
"model": "baai/bge-reranker-v2-m3",
"query": "European capital",
"documents": [
"Paris is in France.",
"A recipe for soup."
],
"top_n": 1
}`,
});
console.log(await response.text());
python
import os
import urllib.request
body = (r'''{
"model": "baai/bge-reranker-v2-m3",
"query": "European capital",
"documents": [
"Paris is in France.",
"A recipe for soup."
],
"top_n": 1
}''').encode("utf-8")
request = urllib.request.Request(
"https://api.tokenlane.de/v1/rerank",
method="POST",
headers={
"Authorization": "Bearer " + os.environ["TOKENLANE_API_KEY"],
"Content-Type": "application/json",
},
data=body,
)
with urllib.request.urlopen(request) as response:
print(response.read().decode("utf-8"))
vercel
// This operation uses HTTP directly.
const response = await fetch("https://api.tokenlane.de/v1/rerank", {
method: "POST",
headers: {
"Authorization": "Bearer " + process.env.TOKENLANE_API_KEY,
"Content-Type": "application/json",
},
body: `{
"model": "baai/bge-reranker-v2-m3",
"query": "European capital",
"documents": [
"Paris is in France.",
"A recipe for soup."
],
"top_n": 1
}`,
});
console.log(await response.text());
Each result has the index of the document in your list, a relevance_score and the document you sent. The list is sorted best first.
Request format
| Field | Value |
|---|---|
model | Required. A reranking model. A chat model fails with 404 no_endpoint. |
query | Required. Text, at most 65,536 UTF-8 bytes, not blank. |
documents | Required. 1 to 1,000 items. Each is a string or an object with a non-empty text string, at most 65,536 UTF-8 bytes. Query and documents together at most 4 MiB of text. |
top_n | Integer from 1 up to the number of documents. Default: all documents. |
return_documents | Boolean. Accepted for compatibility and has no effect: every result carries document, whether the field is omitted, true or false. null or another type fails with 400. |
provider | Routing, as for chat. |
What the provider receives and what comes back
The response always contains your documents, as the OpenRouter rerank format requires a document in every result. There is no option for index and score only.
| Part of your request | Sent to the provider | Returned to you |
|---|---|---|
query | Yes, unchanged. | No. |
| Text of each document | Yes, all documents in your order, also with top_n. | Yes, in results[].document.text, for the returned results only. |
| Other fields of a document object (your ID, title, nested metadata) | No. | Yes, unchanged in results[].document. |
top_n | No. The gateway ranks all documents and then keeps the best top_n. | Only that many results. |
return_documents | No. | No effect. |
model | The provider's model ID for the route. | The public model ID in model. |
A string document comes back as an object with text. An object comes back with all its fields. Put your own ID into a document object, and you do not have to map index back:
"documents": [{ "text": "Madrid is in Spain.", "id": "doc-9", "source": "wiki" }]
The matching result:
{ "document": { "id": "doc-9", "source": "wiki", "text": "Madrid is in Spain." }, "index": 0, "relevance_score": 0.91 }
Everything in document is your own input; nothing else is added. If you pass the results on to a browser, a log or another service, remove document or the fields that must not travel with it.
Response fields:
| Field | Meaning |
|---|---|
results[].index | Zero-based position in your documents. |
results[].relevance_score | A number from 0 to 1. Sorted descending; equal scores keep your order. |
results[].document | Always. The document you sent: a string as { "text": … }, an object with all its fields. |
usage | prompt_tokens, total_tokens, estimated, cost (EUR). |
id, model, provider | Generation ID (also in x-generation-id), the model and the provider that answered. |
The gateway checks the provider's ranking: scores outside 0 to 1, a missing or repeated index or a malformed answer become 502 provider_invalid_response, and the next provider is tried.
Each pair of query and document must fit the model's context (4 bytes of text per token are estimated). With a context of 8,192 tokens that is about 32 KB per pair; a longer pair fails with 400 context_length_exceeded.
Choosing a model
Choose a model that serves the rerank endpoint. In the catalog it takes text as input, shows a price per request (pricing.request) and lists no sampling parameters. Compare the context length (the longest query-plus-document pair) and the price. A chat model is not a reranker, and a reranker cannot chat.
Costs
Reranking has no output. The price depends on the route: a fixed price per request (pricing.request), a price per input token, or both. The charge is in usage.cost (EUR), split in cost_details. If a provider reports no token usage, the token count is an estimate (usage.estimated: true), allowed only for routes with a fixed request price; the cost is exact.
With a fixed price per request, one search over 100 candidates costs one request price, and 1,000 searches cost 1,000 request prices, however many documents each one has. Current prices are on each model page in the catalog.
On a fixed-price route, send all candidates in one request instead of several small ones. The reservation before the call is the request price plus the cost of the estimated input tokens at the route's prompt price.
Common errors
| Status, type | Cause and fix |
|---|---|
400 invalid_parameter | Blank query, no or more than 1,000 documents, an object without text, text over 65,536 bytes, top_n below 1 or above the document count, a non-boolean return_documents. Fix: Fix each path in issues. |
400 context_length_exceeded | A query-document pair is longer than the model's context. Fix: Shorten the documents. |
404 no_endpoint | The model is not a reranker. Fix: Choose a model from remedy_hint. |
502 provider_invalid_response | The provider returned an invalid ranking. Fix: Retry; another provider is tried automatically. |
402 insufficient_credits, 429 | Credits or limits. Fix: As for chat. |
Best practice
- Retrieve first, rerank second: 50 to 200 candidates from the vector search, then
top_nof 3 to 10 for the prompt. - Keep documents to one idea each, a few hundred tokens. Long pages dilute the score and use context.
- A
relevance_scoreranks documents for one query. It is not a probability and differs between models; compare scores only within one response, and pick any cut-off by testing on your own data. - Every result carries its document text, so the response grows with the documents. With long documents, keep
top_nsmall;return_documents: falsedoes not shorten the response. - Remove duplicates before you rerank; they take the places of better documents.
- Pass the documents to the chat model in the new order, with their IDs, so that answers can cite sources.
For AI agents
- Endpoint: [
POST /rerank](https://tokenlane.de/docs/gateway/post-L3JlcmFuaw.md?lang=en). - Required:
model(a reranking model fromGET /v1/models),query(string),documents(1 to 1000 strings or objects withtext). - Optional:
top_n(integer from 1 to the document count, applied by the gateway),return_documents(boolean, no effect),provider. - Response:
results[]sorted byrelevance_score(0 to 1, descending) withindex(position indocuments) anddocument(always: your object unchanged, or{ "text": … }for a string);usage.cost(EUR). - Data flow: only
queryand the document texts reach the provider; object fields,top_nandreturn_documentsstay in the gateway. - Limits: 1000 documents; 65,536 UTF-8 bytes per text; 4 MiB in total; each query-document pair within the model's context; no streaming.
- Errors: 400
invalid_parameterandcontext_length_exceeded, 404no_endpoint, 502provider_invalid_response, 402, 429.