List benchmarks
Vendor data is empty because no licensed vendor-source snapshots are imported. The separate lmarena extension retains published CC BY 4.0 observations, attribution, source variants, dates, confidence intervals and population sizes. Elo and agent IPS stay separate. Only canonical models appear here.
Filter by source, arena, category and max_results. Unavailable sources return an explained empty result. Unsupported task or benchmark filters appear in x-ignored-params. Legacy underscore source names are accepted. No API key is required.
GET https://api.tokenlane.de/v1/benchmarks
Authorizations
[]
curl
curl --request GET \
--url 'https://api.tokenlane.de/v1/benchmarks'
node
const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
method: "GET",
headers: {
},
});
console.log(await response.text());
python
import os
import urllib.request
request = urllib.request.Request(
"https://api.tokenlane.de/v1/benchmarks",
method="GET",
headers={
},
)
with urllib.request.urlopen(request) as response:
print(response.read().decode("utf-8"))
vercel
// This operation uses HTTP directly.
const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
method: "GET",
headers: {
},
});
console.log(await response.text());
query: source (optional)
lmarena returns licensed data; artificial-analysis, design-arena and openrouter return an explained empty result
s7: string
{}
query: arena (optional)
Exact source arena, such as text, vision, webdev or agent; omitted returns all arenas
query: category (optional)
Exact source category; omitted returns all categories
query: max_results (optional)
Positive maximum number of observations; omitted returns all matching bounded snapshot observations
s8: integer
{"minimum":"1"}
query: task_type (optional)
Accepted but not evaluated; reported in x-ignored-params because source task taxonomies are not interchangeable
query: benchmark_type (optional)
Accepted but not evaluated; reported in x-ignored-params because OpenRouter eval results are not imported
Response 200: application/json
Vendor benchmark envelope and separately attributed licensed LMArena extension
{
"data": [],
"lmarena": {
"data": [
{
"arena": "text",
"as_of": "2026-10-27",
"category": "overall",
"display_name": "mock/echo",
"elo": 1200,
"elo_lower": 1190,
"elo_upper": 1210,
"fetched_at": "2026-10-27T12:00:00Z",
"metric": "bradley_terry",
"model_permaslug": "mock/echo",
"primary_variant": true,
"rank": 1,
"rank_of": 100,
"source": "lmarena",
"source_name": "synthetic-model",
"source_row": 0,
"votes": 12000
}
],
"meta": {
"as_of": "2026-10-27T12:00:00Z",
"citation": "Source: LMArena (lmarena.ai), CC BY 4.0",
"model_count": 1,
"note": "Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate.",
"source": null,
"source_url": "https://lmarena.ai",
"task_type": null,
"version": "v1"
}
},
"meta": {
"as_of": "",
"citation": null,
"model_count": 0,
"source": null,
"source_url": null,
"task_type": null,
"version": "v1"
}
}
s9: object
Vendor benchmark envelope plus separately attributed licensed LMArena observations.
{}
- data (required): s272
s272: array
Vendor source observations; empty because no corresponding licensed dataset is imported.
{}
s228: oneOf
The four vendor item forms. This gateway currently imports none of these datasets.
{}
oneOf: s1044
s1044: object
Artificial Analysis composite indices.
{}
- agentic_index (required): s1011
s1011: number | null
Published agentic index, null if unavailable.
double
{}
- coding_index (required): s1012
s1012: number | null
Published coding index, null if unavailable.
double
{}
- display_name (required): s1013
s1013: string
Source display label.
{}
- intelligence_index (required): s1014
s1014: number | null
Published intelligence index, null if unavailable.
double
{}
- model_permaslug (required): s1015
s1015: string
Stable model identifier.
{}
- pricing (required): s1016
s1016: object | null
Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.
{}
- completion (required): s1249
s1249: string
USD price per output token, as an exact decimal string.
{}
- prompt (required): s1250
s1250: string
USD price per input token, as an exact decimal string.
{}
- source (required): s1017
s1017: string
Exact vendor source.
"artificial-analysis"
{}
oneOf: s1045
s1045: object
Design Arena tournament observations, not LMArena Elo.
{}
- arena (required): s619
s619: string
Source arena.
{}
- avg_generation_time_ms (required): s1018
s1018: number | null
Mean generation time, null when unpublished.
double
{}
- category (required): s1019
s1019: string
Source category.
{}
- display_name (required): s1013
Source display label.
- elo (required): s621
s621: number
Source Elo rating.
double
{}
- model_permaslug (required): s1015
Stable model identifier.
- pricing (required): s1020
s1020: object | null
Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.
{}
- completion (required): s1249
USD price per output token, as an exact decimal string.
- prompt (required): s1250
USD price per input token, as an exact decimal string.
- source (required): s1021
s1021: string
Exact vendor source.
"design-arena"
{}
- tournament_stats (required): s1022
s1022: object
Published placement counts.
{}
- first_place (required): s998
s998: integer | null
First-place count, null when unpublished.
int64
{}
- fourth_place (required): s999
s999: integer | null
Fourth-place count, null when unpublished.
int64
{}
- second_place (required): s1000
s1000: integer | null
Second-place count, null when unpublished.
int64
{}
- third_place (required): s1001
s1001: integer | null
Third-place count, null when unpublished.
int64
{}
- total (required): s1002
s1002: integer | null
Total tournament count, null when unpublished.
int64
{}
- win_rate (required): s1023
s1023: number
Win percentage from zero through one hundred.
double
{}
oneOf: s1046
s1046: object
OpenRouter's classic evaluations.
{}
- accuracy (required): s1024
s1024: number
Aggregate accuracy.
double
{}
- accuracy_stddev (required): s1025
s1025: number | null
Accuracy standard deviation, null for a single run.
double
{}
- avg_cost_per_task (required): s1026
s1026: number | null
Exact USD cost per task, null when unpublished.
double
{}
- benchmark_type (required): s1027
s1027: string
Evaluation identifier.
"gpqa_diamond", "tau_bench_verified_airline"
{}
- display_name (required): s1013
Source display label.
- last_run_timestamp (required): s1028
s1028: string
Timestamp of the most recent public run.
{}
- model_permaslug (required): s1015
Stable model identifier.
- source (required): s1029
s1029: string
Exact vendor source.
"openrouter"
{}
- total_tasks (required): s1030
s1030: integer
Number of evaluated tasks.
int64
{}
oneOf: s1047
s1047: object
OpenRouter search evaluations.
{}
- avg_cost_per_task (required): s1026
Exact USD cost per task, null when unpublished.
- avg_latency_per_task_ms (required): s1031
s1031: number | null
Mean latency per task, null when unpublished.
double
{}
- benchmark_type (required): s1032
s1032: string
Search evaluation identifier.
"search_browsecomp", "search_hle", "search_dsqa", "search_widesearch"
{}
- display_name (required): s1013
Source display label.
- last_run_timestamp (required): s1033
s1033: string
Timestamp of the newest qualifying run.
{}
- model_permaslug (required): s1015
Stable model identifier.
- primary_metric (required): s1034
s1034: string
Meaning of primary_score.
"accuracy", "f1_by_item"
{}
- primary_score (required): s1035
s1035: number
Published headline score.
double
{}
- search_engine (required): s1036
s1036: string
Search engine used by the published configuration.
{}
- search_surface (required): s1037
s1037: string
Request surface used by the evaluation.
"server-tool", "plugin"
{}
- source (required): s1038
s1038: string
Exact vendor source.
"openrouter"
{}
- total_tasks (required): s1030
Number of evaluated tasks.
- lmarena (required): s273
s273: object
Licensed LMArena extension, with its own unchanged metrics and attribution.
{}
- data (required): s828
s828: array
Licensed observations in deterministic order, with unchanged source metrics.
{}
s221: allOf
One attributed LMArena observation from the unified benchmark endpoint.
{}
allOf: s1009
s1009: allOf
Source observation, without changing its metric or participant scope.
{}
allOf: s826
s826: oneOf
Unchanged score, interval and observation count in the source metric.
{}
oneOf: s486
s486: object
Human pairwise votes fitted with the source's Bradley-Terry model.
{}
- elo (required): s1125
s1125: number
Published Elo rating, unchanged from the source.
double
{}
1300.5
- elo_lower (required): s1126
s1126: number
Lower confidence bound for this rating.
double
{}
1290
- elo_upper (required): s1127
s1127: number
Upper confidence bound for this rating.
double
{}
1310
- metric (required): s1128
s1128: string
"bradley_terry"
{}
- votes (required): s1129
s1129: integer
Human votes counted by the source.
int64
{"minimum":"0"}
12000
oneOf: s487
s487: object
Source IPS observations, with their own unit and confidence interval.
{}
- metric (required): s1130
s1130: string
"ips"
{}
- observations (required): s1131
s1131: integer
Agent observations counted by the source, not human votes.
int64
{"minimum":"0"}
1200
- score (required): s1132
s1132: number
Published IPS score; not comparable with Elo ratings.
double
{}
0.75
- score_lower (required): s1133
s1133: number
Lower confidence bound for the IPS score.
double
{}
0.7
- score_upper (required): s1134
s1134: number
Upper confidence bound for the IPS score.
double
{}
0.8
- session_count (optional): s1135
s1135: integer | null
Number of source sessions, absent when unpublished.
int64
{"minimum":"0"}
100
allOf: s827
s827: object
{}
- arena (required): s1185
s1185: string
Source arena, such as text, vision, webdev or agent.
{}
"text"
- as_of (required): s1186
s1186: string
Publication date of this board, YYYY-MM-DD.
{}
"2026-10-27"
- category (required): s1187
s1187: string
Source category within the arena; overall is the unfiltered board.
{}
"overall"
- fetched_at (required): s1188
s1188: string
UTC timestamp when the complete source snapshot was imported.
{}
"2026-10-27T12:00:00Z"
- primary_variant (optional): s1189
s1189: boolean
Whether this name is the curated first LMArena mapping, also used by the Web catalog.
{}
true
- rank (optional): s1190
s1190: integer | null
Published rank, null when the source does not publish one.
int32
{"minimum":"0"}
3
- rank_of (required): s1191
s1191: integer
Number of source participants in this board, not merely models in this gateway.
int32
{"minimum":"0"}
100
- source_name (required): s1192
s1192: string
Exact source model variant; variants are never silently substituted.
{}
"synthetic-model"
- source_row (required): s1193
s1193: integer
Stable source row index within the imported snapshot.
int32
{"minimum":"0"}
1
allOf: s1010
s1010: object
{}
- display_name (required): s1246
s1246: string
Canonical display label; source_name below identifies the measured variant.
{}
"mistralai/ministral-8b"
- model_permaslug (required): s1247
s1247: string
Canonical model identifier, excluding gateway aliases.
{}
"mistralai/ministral-8b"
- source (required): s1248
s1248: string
Source discriminator, lmarena; never a third-party source we do not license.
"lmarena"
{}
- meta (required): s829
s829: object
Attribution and result scope, including unavailable-source explanations.
{}
- as_of (required): s494
s494: string
Last complete import timestamp; empty when no matching snapshot is available.
{}
"2026-10-27T12:00:00Z"
- citation (required): s495
s495: string | null
Required citation, null for a requested unavailable source.
{}
"Source: LMArena (lmarena.ai), CC BY 4.0"
- model_count (required): s496
s496: integer
Number of distinct canonical models actually returned.
{"minimum":"0"}
1
- note (required): s497
s497: string
Why the result is empty or which source semantics apply.
{}
"Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate."
- source (required): s498
s498: string | null
Applied source filter for the separately attributed LMArena extension, or null when no source was selected. Unavailable vendor sources retain an explained empty result.
"lmarena", "artificial-analysis", "design-arena", "openrouter", null
{}
- source_url (required): s499
s499: string | null
Public source URL, null for a requested unavailable source.
{}
"https://lmarena.ai"
- task_type (required): s500
s500: string | null
Applied task filter; null because this extension filters by arena and category instead.
{}
- version (required): s501
s501: string
Public response format version.
"v1"
{}
- meta (required): s274
s274: object
Vendor result scope, never counting or relabeling LMArena observations.
{}
- as_of (required): s1048
s1048: string
Last imported vendor observation timestamp; empty when none is available.
{}
- citation (required): s1049
s1049: string | null
Required source attribution, null without an imported vendor dataset.
{}
- model_count (required): s1050
s1050: integer
Number of distinct models in data, not in the LMArena extension.
{"minimum":"0"}
- source (required): s1051
s1051: string | null
Vendor sources accepted as filters; no data from these sources is imported here.
"artificial-analysis", "design-arena", "openrouter", null
{}
- source_url (required): s1052
s1052: string | null
Source URL, null without an imported vendor dataset.
{}
- task_type (required): s1053
s1053: string | null
Applied task filter; null when not evaluated.
{}
- version (required): s1054
s1054: string
Published vendor response format version.
"v1"
{}
Response 400: application/json
Invalid or duplicated benchmark query parameters
{
"error": {
"code": 400,
"message": "Use a positive integer result limit",
"metadata": {
"docs_url": "/docs#description/invalid-parameter",
"issues": [
{
"code": "invalid_value",
"message": "Use a positive integer result limit",
"path": "query.max_results"
}
],
"remedy_hint": "Check the named benchmark query parameters. Elo filters use the curated primary variant; agent IPS is not Elo.",
"request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
"retryable": false,
"type": "invalid_parameter"
}
}
}
s3: object
Every response with an HTTP status of 400 or higher has this shape, OpenRouter's error format. Everything the gateway adds is inside error.metadata, so OpenRouter clients see nothing unfamiliar. See "Errors" in the introduction for all error types.
{}
{
"error": {
"code": 400,
"message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
"metadata": {
"docs_url": "/docs#description/model-not-found",
"issues": [
{
"code": "unknown_model",
"message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
"path": "$.model"
}
],
"remedy_hint": "Did you mean deepseek/deepseek-v4.1-flash?",
"request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
"retryable": false,
"type": "model_not_found"
}
},
"user_id": "org_01k7tm2c4e6g8j0k2m4p6r8t0v"
}
- error (required): s238
s238: object
What went wrong.
{}
{
"code": 400,
"message": "temperature must be at most 2, got 5 (and 1 more)",
"metadata": {
"docs_url": "/docs#description/invalid-parameter",
"issues": [
{
"code": "out_of_range",
"message": "temperature must be at most 2, got 5",
"path": "$.temperature"
},
{
"code": "invalid_value",
"message": "messages[1].role must be one of system, developer, user, assistant, tool, got \"bot\"",
"path": "$.messages[1].role"
}
],
"request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
"retryable": false,
"type": "invalid_parameter"
}
}
- code (required): s633
s633: integer
The HTTP status of the response, as a number.
int32
{"minimum":"0"}
400
- message (required): s634
s634: string
What went wrong, in English. With several problems: the first one and how many more follow; all of them are in metadata.issues. The text may change; match on metadata.type instead.
{}
"temperature must be at most 2, got 5"
- metadata (optional): s635
s635: object
Details for programs: error type, request ID, every problem, a suggested fix and, for provider errors, which provider failed. Always present on gateway errors.
{}
{
"docs_url": "/docs#description/rate-limited",
"provider_error": {
"category": "rate_limited",
"provider_name": "deepinfra",
"retry_after": 2,
"retryable": true,
"status": 429
},
"provider_name": "deepinfra",
"remedy_hint": "Retry after the Retry-After interval, or allow fallbacks to other providers (provider.allow_fallbacks).",
"request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
"retry_after": 2,
"retryable": true,
"type": "rate_limited"
}
- available_providers (optional): s639
s639: array
Providers that serve the model (when provider.only excluded all of them).
{}
[
"deepinfra",
"open-inference"
]
- docs_url (required): s640
s640: string
Link to the section of this error type in the documentation.
{}
"/docs#description/model-not-found"
- failed_routing_step (optional): s641
s641: string
The routing step that left no route (routing errors only).
{}
"Filter by Max Price"
- issues (optional): s642
s642: array
Every problem found in the request, so all can be fixed at once (at most 20); for routing errors one entry per excluded route, for no_provider_available one per failed attempt.
{}
s123: object
One problem in the request, one reason a route was excluded, or one failed attempt.
{}
{
"code": "invalid_url",
"message": "messages[2].content[0].image_url.url must be an http(s) URL or a base64 data URL",
"path": "$.messages[2].content[0].image_url.url"
}
- code (required): s636
s636: string
Kind of problem, stable for programs.
"required", "invalid_type", "invalid_value", "out_of_range", "empty", "invalid_url", "invalid_base64", "unsupported_media_type", "media_type_mismatch", "media_too_large", "too_many_media", "unsupported_media_transport", "media_fetch_failed", "unknown_tool", "unknown_model", "not_available", "not_allowed", "route_excluded", "bridge_failed", "provider_failed"
{}
"out_of_range"
- message (required): s637
s637: string
The problem in English, naming the field and, if short, the value sent.
{}
"temperature must be at most 2, got 5"
- path (required): s638
s638: string
Where the problem is: JSONPath of the field in the request body, for example $.messages[2].content[0].image_url.url, or query.<name> for a query parameter.
{}
"$.temperature"
- provider_error (optional): s643
s643: oneOf
Safe provider diagnostics, identical for HTTP errors and terminal stream events.
{}
oneOf: s1158
s1158: object
Provider diagnostics without provider-authored text. HTTP and terminal SSE errors use this same structure; arbitrary upstream messages and metadata are discarded.
{}
{
"category": "rate_limited",
"provider_name": "deepinfra",
"retry_after": 2,
"retryable": true,
"status": 429
}
- category (required): s1299
s1299: string
Gateway classification from HTTP status and recognized dialect identifiers.
"rate_limited", "overloaded", "invalid_request", "context_length", "content_filtered", "authentication", "billing", "timeout", "unavailable", "unknown"
{}
"rate_limited"
- code (optional): s1300
s1300: string
Exact identifier from the configured dialect's fixed list; arbitrary values vanish.
{}
"rate_limit_exceeded"
- provider_name (required): s1301
s1301: string
Configured provider name, never a name supplied by the upstream body.
{}
"deepinfra"
- retry_after (optional): s1302
s1302: integer
Bounded seconds from the upstream Retry-After header, when present and numeric.
int64
{"minimum":"0"}
2
- retryable (required): s1303
s1303: boolean
Whether the classified failure can be transient. Unknown failures are not promised retryable.
{}
- status (optional): s1304
s1304: integer
Actual HTTP status received from the upstream, including 200 for native errors. Absent when no HTTP status is available.
int32
{"minimum":"0"}
429
oneOf: s588
s588: null
{}
- provider_name (optional): s644
s644: string
The provider that failed, as in the response field provider (provider errors only).
{}
"deepinfra"
- raw (optional): s645
s645: string
Reserved OpenRouter compatibility field. The gateway never fills it: provider free text is deliberately withheld. Use provider_error instead.
{}
""
- remedy_hint (optional): s646
s646: string
A concrete fix, for example the closest model ID to a mistyped one.
{}
"Did you mean deepseek/deepseek-v4.1-flash?"
- request_id (required): s647
s647: string
ID of the request, the same as the x-request-id response header. Quote it when contacting support.
{}
"req_01k7tm9qz3f5c8x2v6b4n0m7hd"
- requested_providers (optional): s648
s648: array
Providers named in provider.only (when it excluded all routes).
{}
[
"scaleway"
]
- retry_after (optional): s649
s649: integer
Seconds to wait before retrying; also sent as the Retry-After header (429 and 503).
int64
{"minimum":"0"}
2
- retryable (required): s650
s650: boolean
Whether sending the identical request again can succeed (for example after a timeout or rate limit), as opposed to errors that need a changed request.
{}
- routing_funnel (optional): s651
s651: array
How routing narrowed down the providers: each step with the number of routes left (routing errors only, same format as OpenRouter).
{}
s200: object
One routing step, as in OpenRouter's routing_funnel.
{}
{
"endpoint_count": 0,
"reason": "provider.max_price",
"step": "Filter by Max Price"
}
- endpoint_count (required): s979
s979: integer
Number of routes (providers of the model) left after the step.
{"minimum":"0"}
2
- reason (optional): s980
s980: string
The request field the step filtered on.
{}
"provider.only"
- step (required): s981
s981: string
Name of the step, as at OpenRouter.
{}
"Initial Endpoints"
- type (required): s652
s652: string
Stable error type to match on in code; see "Errors" in the introduction. Messages may change, the type does not.
"invalid_request", "invalid_parameter", "unsupported_parameter", "context_length_exceeded", "authentication", "insufficient_credits", "workspace_budget_exceeded", "permission_denied", "moderation", "model_not_found", "no_endpoint", "not_found", "method_not_allowed", "timeout", "payload_too_large", "media_fetch_failed", "rate_limited", "provider_error", "idempotency_conflict", "idempotency_in_progress", "idempotency_unavailable", "provider_invalid_response", "no_provider_available", "byok_invalid_credential", "byok_rate_limited", "byok_unavailable", "feature_unavailable", "internal"
{}
"invalid_parameter"
- user_id (optional): s239
s239: string | null
Organization of the API key, if the key was recognized.
{}
"org_01k7tm2c4e6g8j0k2m4p6r8t0v"