# List benchmarks

Vendor `data` is empty because no licensed vendor-source snapshots are imported.
The separate `lmarena` extension retains published CC BY 4.0 observations, attribution,
source variants, dates, confidence intervals and population sizes. Elo and agent IPS
stay separate. Only canonical models appear here.

Filter by source, arena, category and max_results.
Unavailable sources return an explained empty result.
Unsupported task or benchmark filters appear in x-ignored-params.
Legacy underscore source names are accepted. No API key is required.

`GET https://api.tokenlane.de/v1/benchmarks`

## Authorizations

[]

## curl

```curl
curl --request GET \
  --url 'https://api.tokenlane.de/v1/benchmarks'
```

## node

```node
const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
  method: "GET",
  headers: {
  },
});
console.log(await response.text());
```

## python

```python
import os
import urllib.request

request = urllib.request.Request(
    "https://api.tokenlane.de/v1/benchmarks",
    method="GET",
    headers={
    },
)

with urllib.request.urlopen(request) as response:
    print(response.read().decode("utf-8"))
```

## vercel

```vercel
// This operation uses HTTP directly.
const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
  method: "GET",
  headers: {
  },
});
console.log(await response.text());
```

## query: source (optional)

lmarena returns licensed data; artificial-analysis, design-arena and openrouter return an explained empty result

### s7: string

{}

## query: arena (optional)

Exact source arena, such as text, vision, webdev or agent; omitted returns all arenas

## query: category (optional)

Exact source category; omitted returns all categories

## query: max_results (optional)

Positive maximum number of observations; omitted returns all matching bounded snapshot observations

### s8: integer

{"minimum":"1"}

## query: task_type (optional)

Accepted but not evaluated; reported in x-ignored-params because source task taxonomies are not interchangeable

## query: benchmark_type (optional)

Accepted but not evaluated; reported in x-ignored-params because OpenRouter eval results are not imported

## Response 200: application/json

Vendor benchmark envelope and separately attributed licensed LMArena extension

```json
{
  "data": [],
  "lmarena": {
    "data": [
      {
        "arena": "text",
        "as_of": "2026-10-27",
        "category": "overall",
        "display_name": "mock/echo",
        "elo": 1200,
        "elo_lower": 1190,
        "elo_upper": 1210,
        "fetched_at": "2026-10-27T12:00:00Z",
        "metric": "bradley_terry",
        "model_permaslug": "mock/echo",
        "primary_variant": true,
        "rank": 1,
        "rank_of": 100,
        "source": "lmarena",
        "source_name": "synthetic-model",
        "source_row": 0,
        "votes": 12000
      }
    ],
    "meta": {
      "as_of": "2026-10-27T12:00:00Z",
      "citation": "Source: LMArena (lmarena.ai), CC BY 4.0",
      "model_count": 1,
      "note": "Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate.",
      "source": null,
      "source_url": "https://lmarena.ai",
      "task_type": null,
      "version": "v1"
    }
  },
  "meta": {
    "as_of": "",
    "citation": null,
    "model_count": 0,
    "source": null,
    "source_url": null,
    "task_type": null,
    "version": "v1"
  }
}
```

### s9: object

Vendor benchmark envelope plus separately attributed licensed LMArena observations.

{}

- data (required): s272

### s272: array

Vendor source observations; empty because no corresponding licensed dataset is imported.

{}

### s228: oneOf

The four vendor item forms. This gateway currently imports none of these datasets.

{}

oneOf: s1044

### s1044: object

Artificial Analysis composite indices.

{}

- agentic_index (required): s1011

### s1011: number | null

Published agentic index, null if unavailable.

double

{}

- coding_index (required): s1012

### s1012: number | null

Published coding index, null if unavailable.

double

{}

- display_name (required): s1013

### s1013: string

Source display label.

{}

- intelligence_index (required): s1014

### s1014: number | null

Published intelligence index, null if unavailable.

double

{}

- model_permaslug (required): s1015

### s1015: string

Stable model identifier.

{}

- pricing (required): s1016

### s1016: object | null

Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.

{}

- completion (required): s1249

### s1249: string

USD price per output token, as an exact decimal string.

{}

- prompt (required): s1250

### s1250: string

USD price per input token, as an exact decimal string.

{}

- source (required): s1017

### s1017: string

Exact vendor source.

"artificial-analysis"

{}

oneOf: s1045

### s1045: object

Design Arena tournament observations, not LMArena Elo.

{}

- arena (required): s619

### s619: string

Source arena.

{}

- avg_generation_time_ms (required): s1018

### s1018: number | null

Mean generation time, null when unpublished.

double

{}

- category (required): s1019

### s1019: string

Source category.

{}

- display_name (required): s1013

Source display label.

- elo (required): s621

### s621: number

Source Elo rating.

double

{}

- model_permaslug (required): s1015

Stable model identifier.

- pricing (required): s1020

### s1020: object | null

Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.

{}

- completion (required): s1249

USD price per output token, as an exact decimal string.

- prompt (required): s1250

USD price per input token, as an exact decimal string.

- source (required): s1021

### s1021: string

Exact vendor source.

"design-arena"

{}

- tournament_stats (required): s1022

### s1022: object

Published placement counts.

{}

- first_place (required): s998

### s998: integer | null

First-place count, null when unpublished.

int64

{}

- fourth_place (required): s999

### s999: integer | null

Fourth-place count, null when unpublished.

int64

{}

- second_place (required): s1000

### s1000: integer | null

Second-place count, null when unpublished.

int64

{}

- third_place (required): s1001

### s1001: integer | null

Third-place count, null when unpublished.

int64

{}

- total (required): s1002

### s1002: integer | null

Total tournament count, null when unpublished.

int64

{}

- win_rate (required): s1023

### s1023: number

Win percentage from zero through one hundred.

double

{}

oneOf: s1046

### s1046: object

OpenRouter's classic evaluations.

{}

- accuracy (required): s1024

### s1024: number

Aggregate accuracy.

double

{}

- accuracy_stddev (required): s1025

### s1025: number | null

Accuracy standard deviation, null for a single run.

double

{}

- avg_cost_per_task (required): s1026

### s1026: number | null

Exact USD cost per task, null when unpublished.

double

{}

- benchmark_type (required): s1027

### s1027: string

Evaluation identifier.

"gpqa_diamond", "tau_bench_verified_airline"

{}

- display_name (required): s1013

Source display label.

- last_run_timestamp (required): s1028

### s1028: string

Timestamp of the most recent public run.

{}

- model_permaslug (required): s1015

Stable model identifier.

- source (required): s1029

### s1029: string

Exact vendor source.

"openrouter"

{}

- total_tasks (required): s1030

### s1030: integer

Number of evaluated tasks.

int64

{}

oneOf: s1047

### s1047: object

OpenRouter search evaluations.

{}

- avg_cost_per_task (required): s1026

Exact USD cost per task, null when unpublished.

- avg_latency_per_task_ms (required): s1031

### s1031: number | null

Mean latency per task, null when unpublished.

double

{}

- benchmark_type (required): s1032

### s1032: string

Search evaluation identifier.

"search_browsecomp", "search_hle", "search_dsqa", "search_widesearch"

{}

- display_name (required): s1013

Source display label.

- last_run_timestamp (required): s1033

### s1033: string

Timestamp of the newest qualifying run.

{}

- model_permaslug (required): s1015

Stable model identifier.

- primary_metric (required): s1034

### s1034: string

Meaning of primary_score.

"accuracy", "f1_by_item"

{}

- primary_score (required): s1035

### s1035: number

Published headline score.

double

{}

- search_engine (required): s1036

### s1036: string

Search engine used by the published configuration.

{}

- search_surface (required): s1037

### s1037: string

Request surface used by the evaluation.

"server-tool", "plugin"

{}

- source (required): s1038

### s1038: string

Exact vendor source.

"openrouter"

{}

- total_tasks (required): s1030

Number of evaluated tasks.

- lmarena (required): s273

### s273: object

Licensed LMArena extension, with its own unchanged metrics and attribution.

{}

- data (required): s828

### s828: array

Licensed observations in deterministic order, with unchanged source metrics.

{}

### s221: allOf

One attributed LMArena observation from the unified benchmark endpoint.

{}

allOf: s1009

### s1009: allOf

Source observation, without changing its metric or participant scope.

{}

allOf: s826

### s826: oneOf

Unchanged score, interval and observation count in the source metric.

{}

oneOf: s486

### s486: object

Human pairwise votes fitted with the source's Bradley-Terry model.

{}

- elo (required): s1125

### s1125: number

Published Elo rating, unchanged from the source.

double

{}

```json
1300.5
```

- elo_lower (required): s1126

### s1126: number

Lower confidence bound for this rating.

double

{}

```json
1290
```

- elo_upper (required): s1127

### s1127: number

Upper confidence bound for this rating.

double

{}

```json
1310
```

- metric (required): s1128

### s1128: string

"bradley_terry"

{}

- votes (required): s1129

### s1129: integer

Human votes counted by the source.

int64

{"minimum":"0"}

```json
12000
```

oneOf: s487

### s487: object

Source IPS observations, with their own unit and confidence interval.

{}

- metric (required): s1130

### s1130: string

"ips"

{}

- observations (required): s1131

### s1131: integer

Agent observations counted by the source, not human votes.

int64

{"minimum":"0"}

```json
1200
```

- score (required): s1132

### s1132: number

Published IPS score; not comparable with Elo ratings.

double

{}

```json
0.75
```

- score_lower (required): s1133

### s1133: number

Lower confidence bound for the IPS score.

double

{}

```json
0.7
```

- score_upper (required): s1134

### s1134: number

Upper confidence bound for the IPS score.

double

{}

```json
0.8
```

- session_count (optional): s1135

### s1135: integer | null

Number of source sessions, absent when unpublished.

int64

{"minimum":"0"}

```json
100
```

allOf: s827

### s827: object

{}

- arena (required): s1185

### s1185: string

Source arena, such as text, vision, webdev or agent.

{}

```json
"text"
```

- as_of (required): s1186

### s1186: string

Publication date of this board, YYYY-MM-DD.

{}

```json
"2026-10-27"
```

- category (required): s1187

### s1187: string

Source category within the arena; overall is the unfiltered board.

{}

```json
"overall"
```

- fetched_at (required): s1188

### s1188: string

UTC timestamp when the complete source snapshot was imported.

{}

```json
"2026-10-27T12:00:00Z"
```

- primary_variant (optional): s1189

### s1189: boolean

Whether this name is the curated first LMArena mapping, also used by the Web catalog.

{}

```json
true
```

- rank (optional): s1190

### s1190: integer | null

Published rank, null when the source does not publish one.

int32

{"minimum":"0"}

```json
3
```

- rank_of (required): s1191

### s1191: integer

Number of source participants in this board, not merely models in this gateway.

int32

{"minimum":"0"}

```json
100
```

- source_name (required): s1192

### s1192: string

Exact source model variant; variants are never silently substituted.

{}

```json
"synthetic-model"
```

- source_row (required): s1193

### s1193: integer

Stable source row index within the imported snapshot.

int32

{"minimum":"0"}

```json
1
```

allOf: s1010

### s1010: object

{}

- display_name (required): s1246

### s1246: string

Canonical display label; source_name below identifies the measured variant.

{}

```json
"mistralai/ministral-8b"
```

- model_permaslug (required): s1247

### s1247: string

Canonical model identifier, excluding gateway aliases.

{}

```json
"mistralai/ministral-8b"
```

- source (required): s1248

### s1248: string

Source discriminator, lmarena; never a third-party source we do not license.

"lmarena"

{}

- meta (required): s829

### s829: object

Attribution and result scope, including unavailable-source explanations.

{}

- as_of (required): s494

### s494: string

Last complete import timestamp; empty when no matching snapshot is available.

{}

```json
"2026-10-27T12:00:00Z"
```

- citation (required): s495

### s495: string | null

Required citation, null for a requested unavailable source.

{}

```json
"Source: LMArena (lmarena.ai), CC BY 4.0"
```

- model_count (required): s496

### s496: integer

Number of distinct canonical models actually returned.

{"minimum":"0"}

```json
1
```

- note (required): s497

### s497: string

Why the result is empty or which source semantics apply.

{}

```json
"Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate."
```

- source (required): s498

### s498: string | null

Applied source filter for the separately attributed LMArena extension, or null when no source was selected. Unavailable vendor sources retain an explained empty result.

"lmarena", "artificial-analysis", "design-arena", "openrouter", null

{}

- source_url (required): s499

### s499: string | null

Public source URL, null for a requested unavailable source.

{}

```json
"https://lmarena.ai"
```

- task_type (required): s500

### s500: string | null

Applied task filter; null because this extension filters by arena and category instead.

{}

- version (required): s501

### s501: string

Public response format version.

"v1"

{}

- meta (required): s274

### s274: object

Vendor result scope, never counting or relabeling LMArena observations.

{}

- as_of (required): s1048

### s1048: string

Last imported vendor observation timestamp; empty when none is available.

{}

- citation (required): s1049

### s1049: string | null

Required source attribution, null without an imported vendor dataset.

{}

- model_count (required): s1050

### s1050: integer

Number of distinct models in data, not in the LMArena extension.

{"minimum":"0"}

- source (required): s1051

### s1051: string | null

Vendor sources accepted as filters; no data from these sources is imported here.

"artificial-analysis", "design-arena", "openrouter", null

{}

- source_url (required): s1052

### s1052: string | null

Source URL, null without an imported vendor dataset.

{}

- task_type (required): s1053

### s1053: string | null

Applied task filter; null when not evaluated.

{}

- version (required): s1054

### s1054: string

Published vendor response format version.

"v1"

{}

## Response 400: application/json

Invalid or duplicated benchmark query parameters

```json
{
  "error": {
    "code": 400,
    "message": "Use a positive integer result limit",
    "metadata": {
      "docs_url": "/docs#description/invalid-parameter",
      "issues": [
        {
          "code": "invalid_value",
          "message": "Use a positive integer result limit",
          "path": "query.max_results"
        }
      ],
      "remedy_hint": "Check the named benchmark query parameters. Elo filters use the curated primary variant; agent IPS is not Elo.",
      "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
      "retryable": false,
      "type": "invalid_parameter"
    }
  }
}
```

### s3: object

Every response with an HTTP status of 400 or higher has this shape, OpenRouter's error
format. Everything the gateway adds is inside `error.metadata`, so OpenRouter clients see
nothing unfamiliar. See "Errors" in the introduction for all error types.

{}

```json
{
  "error": {
    "code": 400,
    "message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
    "metadata": {
      "docs_url": "/docs#description/model-not-found",
      "issues": [
        {
          "code": "unknown_model",
          "message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
          "path": "$.model"
        }
      ],
      "remedy_hint": "Did you mean deepseek/deepseek-v4.1-flash?",
      "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
      "retryable": false,
      "type": "model_not_found"
    }
  },
  "user_id": "org_01k7tm2c4e6g8j0k2m4p6r8t0v"
}
```

- error (required): s238

### s238: object

What went wrong.

{}

```json
{
  "code": 400,
  "message": "temperature must be at most 2, got 5 (and 1 more)",
  "metadata": {
    "docs_url": "/docs#description/invalid-parameter",
    "issues": [
      {
        "code": "out_of_range",
        "message": "temperature must be at most 2, got 5",
        "path": "$.temperature"
      },
      {
        "code": "invalid_value",
        "message": "messages[1].role must be one of system, developer, user, assistant, tool, got \"bot\"",
        "path": "$.messages[1].role"
      }
    ],
    "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
    "retryable": false,
    "type": "invalid_parameter"
  }
}
```

- code (required): s633

### s633: integer

The HTTP status of the response, as a number.

int32

{"minimum":"0"}

```json
400
```

- message (required): s634

### s634: string

What went wrong, in English. With several problems: the first one and how many more
follow; all of them are in `metadata.issues`. The text may change; match on
`metadata.type` instead.

{}

```json
"temperature must be at most 2, got 5"
```

- metadata (optional): s635

### s635: object

Details for programs: error type, request ID, every problem, a suggested fix and, for
provider errors, which provider failed. Always present on gateway errors.

{}

```json
{
  "docs_url": "/docs#description/rate-limited",
  "provider_error": {
    "category": "rate_limited",
    "provider_name": "deepinfra",
    "retry_after": 2,
    "retryable": true,
    "status": 429
  },
  "provider_name": "deepinfra",
  "remedy_hint": "Retry after the Retry-After interval, or allow fallbacks to other providers (provider.allow_fallbacks).",
  "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
  "retry_after": 2,
  "retryable": true,
  "type": "rate_limited"
}
```

- available_providers (optional): s639

### s639: array

Providers that serve the model (when `provider.only` excluded all of them).

{}

```json
[
  "deepinfra",
  "open-inference"
]
```

- docs_url (required): s640

### s640: string

Link to the section of this error type in the documentation.

{}

```json
"/docs#description/model-not-found"
```

- failed_routing_step (optional): s641

### s641: string

The routing step that left no route (routing errors only).

{}

```json
"Filter by Max Price"
```

- issues (optional): s642

### s642: array

Every problem found in the request, so all can be fixed at once (at most 20); for
routing errors one entry per excluded route, for `no_provider_available` one per failed
attempt.

{}

### s123: object

One problem in the request, one reason a route was excluded, or one failed attempt.

{}

```json
{
  "code": "invalid_url",
  "message": "messages[2].content[0].image_url.url must be an http(s) URL or a base64 data URL",
  "path": "$.messages[2].content[0].image_url.url"
}
```

- code (required): s636

### s636: string

Kind of problem, stable for programs.

"required", "invalid_type", "invalid_value", "out_of_range", "empty", "invalid_url", "invalid_base64", "unsupported_media_type", "media_type_mismatch", "media_too_large", "too_many_media", "unsupported_media_transport", "media_fetch_failed", "unknown_tool", "unknown_model", "not_available", "not_allowed", "route_excluded", "bridge_failed", "provider_failed"

{}

```json
"out_of_range"
```

- message (required): s637

### s637: string

The problem in English, naming the field and, if short, the value sent.

{}

```json
"temperature must be at most 2, got 5"
```

- path (required): s638

### s638: string

Where the problem is: JSONPath of the field in the request body, for example
`$.messages[2].content[0].image_url.url`, or `query.<name>` for a query parameter.

{}

```json
"$.temperature"
```

- provider_error (optional): s643

### s643: oneOf

Safe provider diagnostics, identical for HTTP errors and terminal stream events.

{}

oneOf: s1158

### s1158: object

Provider diagnostics without provider-authored text. HTTP and terminal SSE errors use
this same structure; arbitrary upstream messages and metadata are discarded.

{}

```json
{
  "category": "rate_limited",
  "provider_name": "deepinfra",
  "retry_after": 2,
  "retryable": true,
  "status": 429
}
```

- category (required): s1299

### s1299: string

Gateway classification from HTTP status and recognized dialect identifiers.

"rate_limited", "overloaded", "invalid_request", "context_length", "content_filtered", "authentication", "billing", "timeout", "unavailable", "unknown"

{}

```json
"rate_limited"
```

- code (optional): s1300

### s1300: string

Exact identifier from the configured dialect's fixed list; arbitrary values vanish.

{}

```json
"rate_limit_exceeded"
```

- provider_name (required): s1301

### s1301: string

Configured provider name, never a name supplied by the upstream body.

{}

```json
"deepinfra"
```

- retry_after (optional): s1302

### s1302: integer

Bounded seconds from the upstream Retry-After header, when present and numeric.

int64

{"minimum":"0"}

```json
2
```

- retryable (required): s1303

### s1303: boolean

Whether the classified failure can be transient. Unknown failures are not promised retryable.

{}

- status (optional): s1304

### s1304: integer

Actual HTTP status received from the upstream, including 200 for native errors.
Absent when no HTTP status is available.

int32

{"minimum":"0"}

```json
429
```

oneOf: s588

### s588: null

{}

- provider_name (optional): s644

### s644: string

The provider that failed, as in the response field `provider` (provider errors only).

{}

```json
"deepinfra"
```

- raw (optional): s645

### s645: string

Reserved OpenRouter compatibility field. The gateway never fills it: provider
free text is deliberately withheld. Use provider_error instead.

{}

```json
""
```

- remedy_hint (optional): s646

### s646: string

A concrete fix, for example the closest model ID to a mistyped one.

{}

```json
"Did you mean deepseek/deepseek-v4.1-flash?"
```

- request_id (required): s647

### s647: string

ID of the request, the same as the `x-request-id` response header. Quote it when
contacting support.

{}

```json
"req_01k7tm9qz3f5c8x2v6b4n0m7hd"
```

- requested_providers (optional): s648

### s648: array

Providers named in `provider.only` (when it excluded all routes).

{}

```json
[
  "scaleway"
]
```

- retry_after (optional): s649

### s649: integer

Seconds to wait before retrying; also sent as the `Retry-After` header (429 and 503).

int64

{"minimum":"0"}

```json
2
```

- retryable (required): s650

### s650: boolean

Whether sending the identical request again can succeed (for example after a timeout
or rate limit), as opposed to errors that need a changed request.

{}

- routing_funnel (optional): s651

### s651: array

How routing narrowed down the providers: each step with the number of routes left
(routing errors only, same format as OpenRouter).

{}

### s200: object

One routing step, as in OpenRouter's `routing_funnel`.

{}

```json
{
  "endpoint_count": 0,
  "reason": "provider.max_price",
  "step": "Filter by Max Price"
}
```

- endpoint_count (required): s979

### s979: integer

Number of routes (providers of the model) left after the step.

{"minimum":"0"}

```json
2
```

- reason (optional): s980

### s980: string

The request field the step filtered on.

{}

```json
"provider.only"
```

- step (required): s981

### s981: string

Name of the step, as at OpenRouter.

{}

```json
"Initial Endpoints"
```

- type (required): s652

### s652: string

Stable error type to match on in code; see "Errors" in the introduction. Messages may
change, the type does not.

"invalid_request", "invalid_parameter", "unsupported_parameter", "context_length_exceeded", "authentication", "insufficient_credits", "workspace_budget_exceeded", "permission_denied", "moderation", "model_not_found", "no_endpoint", "not_found", "method_not_allowed", "timeout", "payload_too_large", "media_fetch_failed", "rate_limited", "provider_error", "idempotency_conflict", "idempotency_in_progress", "idempotency_unavailable", "provider_invalid_response", "no_provider_available", "byok_invalid_credential", "byok_rate_limited", "byok_unavailable", "feature_unavailable", "internal"

{}

```json
"invalid_parameter"
```

- user_id (optional): s239

### s239: string | null

Organization of the API key, if the key was recognized.

{}

```json
"org_01k7tm2c4e6g8j0k2m4p6r8t0v"
```
