List benchmarks

Vendor data is empty because no licensed vendor-source snapshots are imported. The separate lmarena extension retains published CC BY 4.0 observations, attribution, source variants, dates, confidence intervals and population sizes. Elo and agent IPS stay separate. Only canonical models appear here.

Filter by source, arena, category and max_results. Unavailable sources return an explained empty result. Unsupported task or benchmark filters appear in x-ignored-params. Legacy underscore source names are accepted. No API key is required.

GET https://api.tokenlane.de/v1/benchmarks

Authorizations

[]

curl

curl --request GET \
  --url 'https://api.tokenlane.de/v1/benchmarks'

node

const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
  method: "GET",
  headers: {
  },
});
console.log(await response.text());

python

import os
import urllib.request

request = urllib.request.Request(
    "https://api.tokenlane.de/v1/benchmarks",
    method="GET",
    headers={
    },
)

with urllib.request.urlopen(request) as response:
    print(response.read().decode("utf-8"))

vercel

// This operation uses HTTP directly.
const response = await fetch("https://api.tokenlane.de/v1/benchmarks", {
  method: "GET",
  headers: {
  },
});
console.log(await response.text());

query: source (optional)

lmarena returns licensed data; artificial-analysis, design-arena and openrouter return an explained empty result

s7: string

{}

query: arena (optional)

Exact source arena, such as text, vision, webdev or agent; omitted returns all arenas

query: category (optional)

Exact source category; omitted returns all categories

query: max_results (optional)

Positive maximum number of observations; omitted returns all matching bounded snapshot observations

s8: integer

{"minimum":"1"}

query: task_type (optional)

Accepted but not evaluated; reported in x-ignored-params because source task taxonomies are not interchangeable

query: benchmark_type (optional)

Accepted but not evaluated; reported in x-ignored-params because OpenRouter eval results are not imported

Response 200: application/json

Vendor benchmark envelope and separately attributed licensed LMArena extension

{
  "data": [],
  "lmarena": {
    "data": [
      {
        "arena": "text",
        "as_of": "2026-10-27",
        "category": "overall",
        "display_name": "mock/echo",
        "elo": 1200,
        "elo_lower": 1190,
        "elo_upper": 1210,
        "fetched_at": "2026-10-27T12:00:00Z",
        "metric": "bradley_terry",
        "model_permaslug": "mock/echo",
        "primary_variant": true,
        "rank": 1,
        "rank_of": 100,
        "source": "lmarena",
        "source_name": "synthetic-model",
        "source_row": 0,
        "votes": 12000
      }
    ],
    "meta": {
      "as_of": "2026-10-27T12:00:00Z",
      "citation": "Source: LMArena (lmarena.ai), CC BY 4.0",
      "model_count": 1,
      "note": "Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate.",
      "source": null,
      "source_url": "https://lmarena.ai",
      "task_type": null,
      "version": "v1"
    }
  },
  "meta": {
    "as_of": "",
    "citation": null,
    "model_count": 0,
    "source": null,
    "source_url": null,
    "task_type": null,
    "version": "v1"
  }
}

s9: object

Vendor benchmark envelope plus separately attributed licensed LMArena observations.

{}

  • data (required): s272

s272: array

Vendor source observations; empty because no corresponding licensed dataset is imported.

{}

s228: oneOf

The four vendor item forms. This gateway currently imports none of these datasets.

{}

oneOf: s1044

s1044: object

Artificial Analysis composite indices.

{}

  • agentic_index (required): s1011

s1011: number | null

Published agentic index, null if unavailable.

double

{}

  • coding_index (required): s1012

s1012: number | null

Published coding index, null if unavailable.

double

{}

  • display_name (required): s1013

s1013: string

Source display label.

{}

  • intelligence_index (required): s1014

s1014: number | null

Published intelligence index, null if unavailable.

double

{}

  • model_permaslug (required): s1015

s1015: string

Stable model identifier.

{}

  • pricing (required): s1016

s1016: object | null

Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.

{}

  • completion (required): s1249

s1249: string

USD price per output token, as an exact decimal string.

{}

  • prompt (required): s1250

s1250: string

USD price per input token, as an exact decimal string.

{}

  • source (required): s1017

s1017: string

Exact vendor source.

"artificial-analysis"

{}

oneOf: s1045

s1045: object

Design Arena tournament observations, not LMArena Elo.

{}

  • arena (required): s619

s619: string

Source arena.

{}

  • avg_generation_time_ms (required): s1018

s1018: number | null

Mean generation time, null when unpublished.

double

{}

  • category (required): s1019

s1019: string

Source category.

{}

  • display_name (required): s1013

Source display label.

  • elo (required): s621

s621: number

Source Elo rating.

double

{}

  • model_permaslug (required): s1015

Stable model identifier.

  • pricing (required): s1020

s1020: object | null

Source-published USD prices per input/output token, or null when unavailable. These are benchmark-source metadata, not customer EUR tariffs.

{}

  • completion (required): s1249

USD price per output token, as an exact decimal string.

  • prompt (required): s1250

USD price per input token, as an exact decimal string.

  • source (required): s1021

s1021: string

Exact vendor source.

"design-arena"

{}

  • tournament_stats (required): s1022

s1022: object

Published placement counts.

{}

  • first_place (required): s998

s998: integer | null

First-place count, null when unpublished.

int64

{}

  • fourth_place (required): s999

s999: integer | null

Fourth-place count, null when unpublished.

int64

{}

  • second_place (required): s1000

s1000: integer | null

Second-place count, null when unpublished.

int64

{}

  • third_place (required): s1001

s1001: integer | null

Third-place count, null when unpublished.

int64

{}

  • total (required): s1002

s1002: integer | null

Total tournament count, null when unpublished.

int64

{}

  • win_rate (required): s1023

s1023: number

Win percentage from zero through one hundred.

double

{}

oneOf: s1046

s1046: object

OpenRouter's classic evaluations.

{}

  • accuracy (required): s1024

s1024: number

Aggregate accuracy.

double

{}

  • accuracy_stddev (required): s1025

s1025: number | null

Accuracy standard deviation, null for a single run.

double

{}

  • avg_cost_per_task (required): s1026

s1026: number | null

Exact USD cost per task, null when unpublished.

double

{}

  • benchmark_type (required): s1027

s1027: string

Evaluation identifier.

"gpqa_diamond", "tau_bench_verified_airline"

{}

  • display_name (required): s1013

Source display label.

  • last_run_timestamp (required): s1028

s1028: string

Timestamp of the most recent public run.

{}

  • model_permaslug (required): s1015

Stable model identifier.

  • source (required): s1029

s1029: string

Exact vendor source.

"openrouter"

{}

  • total_tasks (required): s1030

s1030: integer

Number of evaluated tasks.

int64

{}

oneOf: s1047

s1047: object

OpenRouter search evaluations.

{}

  • avg_cost_per_task (required): s1026

Exact USD cost per task, null when unpublished.

  • avg_latency_per_task_ms (required): s1031

s1031: number | null

Mean latency per task, null when unpublished.

double

{}

  • benchmark_type (required): s1032

s1032: string

Search evaluation identifier.

"search_browsecomp", "search_hle", "search_dsqa", "search_widesearch"

{}

  • display_name (required): s1013

Source display label.

  • last_run_timestamp (required): s1033

s1033: string

Timestamp of the newest qualifying run.

{}

  • model_permaslug (required): s1015

Stable model identifier.

  • primary_metric (required): s1034

s1034: string

Meaning of primary_score.

"accuracy", "f1_by_item"

{}

  • primary_score (required): s1035

s1035: number

Published headline score.

double

{}

  • search_engine (required): s1036

s1036: string

Search engine used by the published configuration.

{}

  • search_surface (required): s1037

s1037: string

Request surface used by the evaluation.

"server-tool", "plugin"

{}

  • source (required): s1038

s1038: string

Exact vendor source.

"openrouter"

{}

  • total_tasks (required): s1030

Number of evaluated tasks.

  • lmarena (required): s273

s273: object

Licensed LMArena extension, with its own unchanged metrics and attribution.

{}

  • data (required): s828

s828: array

Licensed observations in deterministic order, with unchanged source metrics.

{}

s221: allOf

One attributed LMArena observation from the unified benchmark endpoint.

{}

allOf: s1009

s1009: allOf

Source observation, without changing its metric or participant scope.

{}

allOf: s826

s826: oneOf

Unchanged score, interval and observation count in the source metric.

{}

oneOf: s486

s486: object

Human pairwise votes fitted with the source's Bradley-Terry model.

{}

  • elo (required): s1125

s1125: number

Published Elo rating, unchanged from the source.

double

{}

1300.5
  • elo_lower (required): s1126

s1126: number

Lower confidence bound for this rating.

double

{}

1290
  • elo_upper (required): s1127

s1127: number

Upper confidence bound for this rating.

double

{}

1310
  • metric (required): s1128

s1128: string

"bradley_terry"

{}

  • votes (required): s1129

s1129: integer

Human votes counted by the source.

int64

{"minimum":"0"}

12000

oneOf: s487

s487: object

Source IPS observations, with their own unit and confidence interval.

{}

  • metric (required): s1130

s1130: string

"ips"

{}

  • observations (required): s1131

s1131: integer

Agent observations counted by the source, not human votes.

int64

{"minimum":"0"}

1200
  • score (required): s1132

s1132: number

Published IPS score; not comparable with Elo ratings.

double

{}

0.75
  • score_lower (required): s1133

s1133: number

Lower confidence bound for the IPS score.

double

{}

0.7
  • score_upper (required): s1134

s1134: number

Upper confidence bound for the IPS score.

double

{}

0.8
  • session_count (optional): s1135

s1135: integer | null

Number of source sessions, absent when unpublished.

int64

{"minimum":"0"}

100

allOf: s827

s827: object

{}

  • arena (required): s1185

s1185: string

Source arena, such as text, vision, webdev or agent.

{}

"text"
  • as_of (required): s1186

s1186: string

Publication date of this board, YYYY-MM-DD.

{}

"2026-10-27"
  • category (required): s1187

s1187: string

Source category within the arena; overall is the unfiltered board.

{}

"overall"
  • fetched_at (required): s1188

s1188: string

UTC timestamp when the complete source snapshot was imported.

{}

"2026-10-27T12:00:00Z"
  • primary_variant (optional): s1189

s1189: boolean

Whether this name is the curated first LMArena mapping, also used by the Web catalog.

{}

true
  • rank (optional): s1190

s1190: integer | null

Published rank, null when the source does not publish one.

int32

{"minimum":"0"}

3
  • rank_of (required): s1191

s1191: integer

Number of source participants in this board, not merely models in this gateway.

int32

{"minimum":"0"}

100
  • source_name (required): s1192

s1192: string

Exact source model variant; variants are never silently substituted.

{}

"synthetic-model"
  • source_row (required): s1193

s1193: integer

Stable source row index within the imported snapshot.

int32

{"minimum":"0"}

1

allOf: s1010

s1010: object

{}

  • display_name (required): s1246

s1246: string

Canonical display label; source_name below identifies the measured variant.

{}

"mistralai/ministral-8b"
  • model_permaslug (required): s1247

s1247: string

Canonical model identifier, excluding gateway aliases.

{}

"mistralai/ministral-8b"
  • source (required): s1248

s1248: string

Source discriminator, lmarena; never a third-party source we do not license.

"lmarena"

{}

  • meta (required): s829

s829: object

Attribution and result scope, including unavailable-source explanations.

{}

  • as_of (required): s494

s494: string

Last complete import timestamp; empty when no matching snapshot is available.

{}

"2026-10-27T12:00:00Z"
  • citation (required): s495

s495: string | null

Required citation, null for a requested unavailable source.

{}

"Source: LMArena (lmarena.ai), CC BY 4.0"
  • model_count (required): s496

s496: integer

Number of distinct canonical models actually returned.

{"minimum":"0"}

1
  • note (required): s497

s497: string

Why the result is empty or which source semantics apply.

{}

"Unchanged source observations. Named variants, Elo ratings and agent IPS remain separate."
  • source (required): s498

s498: string | null

Applied source filter for the separately attributed LMArena extension, or null when no source was selected. Unavailable vendor sources retain an explained empty result.

"lmarena", "artificial-analysis", "design-arena", "openrouter", null

{}

  • source_url (required): s499

s499: string | null

Public source URL, null for a requested unavailable source.

{}

"https://lmarena.ai"
  • task_type (required): s500

s500: string | null

Applied task filter; null because this extension filters by arena and category instead.

{}

  • version (required): s501

s501: string

Public response format version.

"v1"

{}

  • meta (required): s274

s274: object

Vendor result scope, never counting or relabeling LMArena observations.

{}

  • as_of (required): s1048

s1048: string

Last imported vendor observation timestamp; empty when none is available.

{}

  • citation (required): s1049

s1049: string | null

Required source attribution, null without an imported vendor dataset.

{}

  • model_count (required): s1050

s1050: integer

Number of distinct models in data, not in the LMArena extension.

{"minimum":"0"}

  • source (required): s1051

s1051: string | null

Vendor sources accepted as filters; no data from these sources is imported here.

"artificial-analysis", "design-arena", "openrouter", null

{}

  • source_url (required): s1052

s1052: string | null

Source URL, null without an imported vendor dataset.

{}

  • task_type (required): s1053

s1053: string | null

Applied task filter; null when not evaluated.

{}

  • version (required): s1054

s1054: string

Published vendor response format version.

"v1"

{}

Response 400: application/json

Invalid or duplicated benchmark query parameters

{
  "error": {
    "code": 400,
    "message": "Use a positive integer result limit",
    "metadata": {
      "docs_url": "/docs#description/invalid-parameter",
      "issues": [
        {
          "code": "invalid_value",
          "message": "Use a positive integer result limit",
          "path": "query.max_results"
        }
      ],
      "remedy_hint": "Check the named benchmark query parameters. Elo filters use the curated primary variant; agent IPS is not Elo.",
      "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
      "retryable": false,
      "type": "invalid_parameter"
    }
  }
}

s3: object

Every response with an HTTP status of 400 or higher has this shape, OpenRouter's error format. Everything the gateway adds is inside error.metadata, so OpenRouter clients see nothing unfamiliar. See "Errors" in the introduction for all error types.

{}

{
  "error": {
    "code": 400,
    "message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
    "metadata": {
      "docs_url": "/docs#description/model-not-found",
      "issues": [
        {
          "code": "unknown_model",
          "message": "deepseek/deepseek-v4.1-flsh is not a valid model ID",
          "path": "$.model"
        }
      ],
      "remedy_hint": "Did you mean deepseek/deepseek-v4.1-flash?",
      "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
      "retryable": false,
      "type": "model_not_found"
    }
  },
  "user_id": "org_01k7tm2c4e6g8j0k2m4p6r8t0v"
}
  • error (required): s238

s238: object

What went wrong.

{}

{
  "code": 400,
  "message": "temperature must be at most 2, got 5 (and 1 more)",
  "metadata": {
    "docs_url": "/docs#description/invalid-parameter",
    "issues": [
      {
        "code": "out_of_range",
        "message": "temperature must be at most 2, got 5",
        "path": "$.temperature"
      },
      {
        "code": "invalid_value",
        "message": "messages[1].role must be one of system, developer, user, assistant, tool, got \"bot\"",
        "path": "$.messages[1].role"
      }
    ],
    "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
    "retryable": false,
    "type": "invalid_parameter"
  }
}
  • code (required): s633

s633: integer

The HTTP status of the response, as a number.

int32

{"minimum":"0"}

400
  • message (required): s634

s634: string

What went wrong, in English. With several problems: the first one and how many more follow; all of them are in metadata.issues. The text may change; match on metadata.type instead.

{}

"temperature must be at most 2, got 5"
  • metadata (optional): s635

s635: object

Details for programs: error type, request ID, every problem, a suggested fix and, for provider errors, which provider failed. Always present on gateway errors.

{}

{
  "docs_url": "/docs#description/rate-limited",
  "provider_error": {
    "category": "rate_limited",
    "provider_name": "deepinfra",
    "retry_after": 2,
    "retryable": true,
    "status": 429
  },
  "provider_name": "deepinfra",
  "remedy_hint": "Retry after the Retry-After interval, or allow fallbacks to other providers (provider.allow_fallbacks).",
  "request_id": "req_01k7tm9qz3f5c8x2v6b4n0m7hd",
  "retry_after": 2,
  "retryable": true,
  "type": "rate_limited"
}
  • available_providers (optional): s639

s639: array

Providers that serve the model (when provider.only excluded all of them).

{}

[
  "deepinfra",
  "open-inference"
]
  • docs_url (required): s640

s640: string

Link to the section of this error type in the documentation.

{}

"/docs#description/model-not-found"
  • failed_routing_step (optional): s641

s641: string

The routing step that left no route (routing errors only).

{}

"Filter by Max Price"
  • issues (optional): s642

s642: array

Every problem found in the request, so all can be fixed at once (at most 20); for routing errors one entry per excluded route, for no_provider_available one per failed attempt.

{}

s123: object

One problem in the request, one reason a route was excluded, or one failed attempt.

{}

{
  "code": "invalid_url",
  "message": "messages[2].content[0].image_url.url must be an http(s) URL or a base64 data URL",
  "path": "$.messages[2].content[0].image_url.url"
}
  • code (required): s636

s636: string

Kind of problem, stable for programs.

"required", "invalid_type", "invalid_value", "out_of_range", "empty", "invalid_url", "invalid_base64", "unsupported_media_type", "media_type_mismatch", "media_too_large", "too_many_media", "unsupported_media_transport", "media_fetch_failed", "unknown_tool", "unknown_model", "not_available", "not_allowed", "route_excluded", "bridge_failed", "provider_failed"

{}

"out_of_range"
  • message (required): s637

s637: string

The problem in English, naming the field and, if short, the value sent.

{}

"temperature must be at most 2, got 5"
  • path (required): s638

s638: string

Where the problem is: JSONPath of the field in the request body, for example $.messages[2].content[0].image_url.url, or query.<name> for a query parameter.

{}

"$.temperature"
  • provider_error (optional): s643

s643: oneOf

Safe provider diagnostics, identical for HTTP errors and terminal stream events.

{}

oneOf: s1158

s1158: object

Provider diagnostics without provider-authored text. HTTP and terminal SSE errors use this same structure; arbitrary upstream messages and metadata are discarded.

{}

{
  "category": "rate_limited",
  "provider_name": "deepinfra",
  "retry_after": 2,
  "retryable": true,
  "status": 429
}
  • category (required): s1299

s1299: string

Gateway classification from HTTP status and recognized dialect identifiers.

"rate_limited", "overloaded", "invalid_request", "context_length", "content_filtered", "authentication", "billing", "timeout", "unavailable", "unknown"

{}

"rate_limited"
  • code (optional): s1300

s1300: string

Exact identifier from the configured dialect's fixed list; arbitrary values vanish.

{}

"rate_limit_exceeded"
  • provider_name (required): s1301

s1301: string

Configured provider name, never a name supplied by the upstream body.

{}

"deepinfra"
  • retry_after (optional): s1302

s1302: integer

Bounded seconds from the upstream Retry-After header, when present and numeric.

int64

{"minimum":"0"}

2
  • retryable (required): s1303

s1303: boolean

Whether the classified failure can be transient. Unknown failures are not promised retryable.

{}

  • status (optional): s1304

s1304: integer

Actual HTTP status received from the upstream, including 200 for native errors. Absent when no HTTP status is available.

int32

{"minimum":"0"}

429

oneOf: s588

s588: null

{}

  • provider_name (optional): s644

s644: string

The provider that failed, as in the response field provider (provider errors only).

{}

"deepinfra"
  • raw (optional): s645

s645: string

Reserved OpenRouter compatibility field. The gateway never fills it: provider free text is deliberately withheld. Use provider_error instead.

{}

""
  • remedy_hint (optional): s646

s646: string

A concrete fix, for example the closest model ID to a mistyped one.

{}

"Did you mean deepseek/deepseek-v4.1-flash?"
  • request_id (required): s647

s647: string

ID of the request, the same as the x-request-id response header. Quote it when contacting support.

{}

"req_01k7tm9qz3f5c8x2v6b4n0m7hd"
  • requested_providers (optional): s648

s648: array

Providers named in provider.only (when it excluded all routes).

{}

[
  "scaleway"
]
  • retry_after (optional): s649

s649: integer

Seconds to wait before retrying; also sent as the Retry-After header (429 and 503).

int64

{"minimum":"0"}

2
  • retryable (required): s650

s650: boolean

Whether sending the identical request again can succeed (for example after a timeout or rate limit), as opposed to errors that need a changed request.

{}

  • routing_funnel (optional): s651

s651: array

How routing narrowed down the providers: each step with the number of routes left (routing errors only, same format as OpenRouter).

{}

s200: object

One routing step, as in OpenRouter's routing_funnel.

{}

{
  "endpoint_count": 0,
  "reason": "provider.max_price",
  "step": "Filter by Max Price"
}
  • endpoint_count (required): s979

s979: integer

Number of routes (providers of the model) left after the step.

{"minimum":"0"}

2
  • reason (optional): s980

s980: string

The request field the step filtered on.

{}

"provider.only"
  • step (required): s981

s981: string

Name of the step, as at OpenRouter.

{}

"Initial Endpoints"
  • type (required): s652

s652: string

Stable error type to match on in code; see "Errors" in the introduction. Messages may change, the type does not.

"invalid_request", "invalid_parameter", "unsupported_parameter", "context_length_exceeded", "authentication", "insufficient_credits", "workspace_budget_exceeded", "permission_denied", "moderation", "model_not_found", "no_endpoint", "not_found", "method_not_allowed", "timeout", "payload_too_large", "media_fetch_failed", "rate_limited", "provider_error", "idempotency_conflict", "idempotency_in_progress", "idempotency_unavailable", "provider_invalid_response", "no_provider_available", "byok_invalid_credential", "byok_rate_limited", "byok_unavailable", "feature_unavailable", "internal"

{}

"invalid_parameter"
  • user_id (optional): s239

s239: string | null

Organization of the API key, if the key was recognized.

{}

"org_01k7tm2c4e6g8j0k2m4p6r8t0v"