Image understanding
Ask a vision model about one or more images: formats, limits, costs and errors.
When to use it
Describe photos, read text in screenshots, compare two pictures, extract values from charts or classify images. The image goes into a chat message next to your question. For multi-page documents use PDFs and files; this guide does not cover image generation.
Minimal example
A text part and an image_url part in one user message. The example embeds a 1 by 1 pixel PNG as a data URL, so it needs no image host.
curl
curl --request POST \
--url 'https://api.tokenlane.de/v1/chat/completions' \
--header 'Authorization: Bearer YOUR_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
"model": "deepseek/deepseek-v4.1-flash-1017",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image."
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"
}
}
]
}
],
"max_tokens": 400
}'
node
import OpenAI from 'openai';
const client = new OpenAI({ baseURL: "https://api.tokenlane.de/v1", apiKey: process.env.TOKENLANE_API_KEY });
const response = await client.chat.completions.create({
"model": "deepseek/deepseek-v4.1-flash-1017",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image."
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"
}
}
]
}
],
"max_tokens": 400
}, {});
console.log(response);
python
import os
import json
from openai import OpenAI
client = OpenAI(base_url="https://api.tokenlane.de/v1", api_key=os.environ["TOKENLANE_API_KEY"])
body = json.loads(r'''{
"model": "deepseek/deepseek-v4.1-flash-1017",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image."
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"
}
}
]
}
],
"max_tokens": 400
}''')
response = client.chat.completions.create(
model=body["model"],
messages=body["messages"],
max_tokens=body["max_tokens"]
)
print(response)
vercel
// This operation uses HTTP directly.
const response = await fetch("https://api.tokenlane.de/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": "Bearer " + process.env.TOKENLANE_API_KEY,
"Content-Type": "application/json",
},
body: `{
"model": "deepseek/deepseek-v4.1-flash-1017",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image."
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4//8/AAX+Av4N70a4AAAAAElFTkSuQmCC"
}
}
]
}
],
"max_tokens": 400
}`,
});
console.log(await response.text());
Request format
The URL example below is a publicly reachable JPEG (checked with HTTP 200). Availability can change; inline data remains the reproducible option.
Image hosts may reject automated downloads. Wikimedia requires a descriptive User-Agent with contact information; an image opening in your browser does not prove that the gateway or provider can fetch it. 400 media_fetch_failed can therefore report an upstream HTTP 400.
Use a directly reachable image URL or inline data.
The image part:
{ "type": "image_url", "image_url": { "url": "https://images.unsplash.com/photo-1500530855697-b586d89ba3ee?w=640&fm=jpg&q=80", "detail": "low" } }
| Field | Value |
|---|---|
type | image_url |
image_url.url | An http or https URL with a host, or a base64 data URL: data:image/png;base64,<data> |
image_url.detail | Optional: low, high or auto. Passed to the provider as sent; a provider may ignore it. |
Send several images as several parts in one message. The gateway keeps their order. It validates the shape before any provider is called. The gateway can fetch a remote URL and adapt the media to the selected route; remote fetches have safety and size limits. A problem names the field, for example $.messages[0].content[1].image_url.url.
One limit applies everywhere: the request body may have 50 MiB, so keep inline images small.
All other limits depend on the route. Each provider sets the accepted types, the size per image, the total size, the number of images, and whether it downloads URLs or takes only inline data. The gateway checks them before the call and skips a route that does not fit. Most routes take inline data only, and PNG and JPEG are accepted everywhere. An image that stays within these bounds fits every route today:
| Safe on all routes | |
|---|---|
| Type | PNG or JPEG |
| Size | at most 3,000,000 bytes decoded, 4,000,000 characters as data URL |
| Count | at most 4 images, together at most 3,000,000 bytes |
| Transport | inline as data URL |
Routes of one model can differ. The exact limits of the route that rejected a request are in issues.
Choosing a model
Choose a model whose architecture.input_modalities contains image. A text-only model fails with 404 no_endpoint ("No endpoints found that support image input"); remedy_hint lists models that accept images. A model can have several routes with different media rules; pin one with provider.only if you need a particular one (see Routing). Providers and jurisdiction per route are on the model page in the catalog.
Costs
An image is billed as input tokens at the prompt price. How many tokens it counts is the provider's decision (it depends on the pixel size), so read usage.prompt_tokens and usage.prompt_tokens_details.image_tokens after the call. A route with a separate image price shows it as pricing.image or pricing.image_token in the catalog.
For inline images with readable dimensions and a verified token formula in the route profile, the gateway estimates the prompt reservation from text and image patches, a fixed allowance where configured, and a safety factor. Scaling is used only when the profile explicitly specifies it.
For example, a 1,540 × 1,540 image with 28 × 28 patches and one token per patch needs 3,025 image tokens; a 10% safety factor reserves 3,328. The output allowance is reserved separately.
The actual provider usage determines the charge; unused reservation is released, and actual usage can exceed the estimate.
Without a formula, outside its verified dimensions unless explicit downscaling applies, or with unknown dimensions or remote URLs, the gateway retains the full context-window reservation. Lowering max_tokens reduces only the output allowance. A key limit or organization balance below the reservation produces 402 insufficient_credits; remedy_hint identifies the method. Invalid image headers still follow the validation rules above.
Common errors
| Status, type | Cause and fix |
|---|---|
404 no_endpoint | The model has no image input. Fix: Take a model from remedy_hint. |
400 invalid_parameter, issue invalid_url | Not an http(s) URL with host, not a base64 data URL. Fix: Send a full URL or data:<type>;base64,<data>. |
400 invalid_parameter, issue invalid_base64 | The data URL does not contain valid base64. Fix: Encode the file bytes without line breaks. |
400 invalid_parameter, failed_routing_step "Filter by Media Limits" | Every route rejects the media. Issue codes per route: unsupported_media_type, media_type_mismatch (bytes do not match the declared type), media_too_large, too_many_media, unsupported_media_transport (URL or inline data not accepted, or HTTPS required). Fix: Convert or shrink the image, send it inline, or choose another model. |
413 payload_too_large | Body above 50 MiB. Fix: Shrink the images. |
402 insufficient_credits | The estimated image allowance or fallback context-window reservation exceeds the key limit or credits. Fix: Read remedy_hint; top up credits, raise the limit, or choose a suitable route. |
400 media_fetch_failed | The gateway could not fetch the remote image safely, including a source returning HTTP 400. Fix: Use a directly reachable URL or inline data; check the host’s bot policy. |
400 invalid_parameter or 502 provider_error | The provider rejected the image or could not fetch it. Provider text is withheld. Fix: Send the image inline; check type and size. |
Best practice
- Resize before sending: most models scale large images down, so extra pixels cost bandwidth and tokens without adding detail. A longest side of about 1,500 to 2,000 pixels is enough unless you need fine print.
- PNG for screenshots and text, JPEG for photos.
- Send private or local images inline. URLs must be reachable by the provider without credentials; never put secrets into a URL, and note that signed URLs expire.
- Ask one clear question per image set and say what to return (a list, a number, JSON).
- Text inside an image is untrusted input. It can contain instructions; do not let the answer trigger actions without your own checks.
- Use a low
detailfor thumbnails and previews if the provider supports it.
For AI agents
- Endpoint: [
POST /chat/completions](https://tokenlane.de/docs/gateway/post-L2NoYXQvY29tcGxldGlvbnM.md?lang=en); the image sits inmessages[].content[]. - Part:
{"type":"image_url","image_url":{"url":"<https URL or data:image/png;base64,...>","detail":"low|high|auto"}}. - Required: a model with
imageinarchitecture.input_modalities(copy the ID fromGET /v1/models). - Limits: body 50 MiB; safe on all routes are PNG or JPEG, inline, up to 4 images, 3,000,000 bytes decoded each and in total.
- Cost: actual input tokens (
usage.prompt_tokens_details.image_tokens). Reservation: verified profile formula with dimensions and safety; otherwise the full context window. - Errors: 404
no_endpoint(no image input), 400invalid_parameterwith issue codesinvalid_url,invalid_base64,unsupported_media_type,media_type_mismatch,media_too_large,too_many_media,unsupported_media_transport, 402insufficient_credits.