Blog / Model watch

How to Verify an API Relay's Model Is Real: 5 Consistency Checks

2026-10-18cdnping editors9 min read

No method can confirm with 100% certainty which model a relay is really serving. You only see requests and responses, and the model field, error messages, and even the answer to "who are you?" can all be rewritten by the middle layer. What you can do is run consistency checks: send the same requests to the official API and to the relay, then compare token counts, context limits, tool-call format, latency and throughput, and the output distribution for fixed prompts. If all of them line up, a swap is very unlikely. If one is clearly off, it is worth digging into.

(In China, "relay" services, also called API proxies or gateways, resell access to model APIs, often at lower prices or with local payment. The checks below apply to any relay or reseller.)

#Why can't you just ask the model "who are you"?

A model's self-introduction comes from training data and the system prompt, not from identity information read at runtime. Common situations:

• The relay can write "You are model X" in the system prompt, and most models will repeat it.
• Many models were trained on corpora containing other models' conversations, so they sometimes name themselves wrongly.
• The model field in the response JSON is filled in by the server, so a relay only has to change one string.

So asking the model only rules out the crudest substitutions. The useful signals come from things a model can't easily fake: the tokenizer, the context limit, protocol details, and inference speed.

#What kinds of "inconsistency" do relays show?

Separate the problem types first so the checks have a direction. When readers say a relay "doesn't feel like the official API", it is usually one of four things, and only the first is a true swap:

The second case is very common with open-weight models, and it isn't always intentional. In the OpenRouter public-data snapshot that cdnping collected on 2026-10-17 (a single collection on that day; data at /en/models), one lightweight DeepSeek model id had 36 provider endpoints: 17 labeled fp8, 3 fp4, 1 mxfp4, and 15 with no precision label. One new Kimi model id had 26 endpoints: 9 fp4, 7 mxfp4, 2 fp8, and 8 unlabeled. The same model name can sit on top of deployments with very different precision. For how quantization affects output and how to pin precision, see Did the model get dumber or did the provider change? Inference providers and fp8/fp4 quantization.

Closed models are different. In the same snapshot, the endpoint precision of every Anthropic, OpenAI, Google, and xAI model is "unknown", and the only providers are the vendor itself and a few large cloud platforms. Closed-model weights aren't released, so third parties can't deploy them on their own. If a relay claims to offer a closed model, then no matter how many hops are in between, the actual inference can only come from the vendor or an authorized cloud platform. That is the starting point for the checks.

#How do you run the five consistency checks?

Setup: get an official API key as the baseline (the checks below use few requests), plus the relay's key. The commands below use the OpenAI-compatible format; replace <model id> with the model you want to verify. For configuring a relay's endpoint address, see Custom API endpoints for Claude Code, Codex, and OpenCode.

First, a small helper function that you'll reuse:

# Usage: ask <BASE_URL> <API_KEY> <JSON request body file>
ask() {
  curl -s "$1/v1/chat/completions" \
    -H "Authorization: Bearer $2" \
    -H "Content-Type: application/json" \
    -d @"$3"
}

#Check 1: Does the input token count match?

Different model families use different tokenizers, so the same text splits into a different number of tokens. This is the hardest signal to fake, because the relay would have to reimplement the target model's tokenizer.

Steps:

1. Prepare a fixed text, ideally a mix of Chinese and English, code, and emoji, about a thousand words long, and save it as probe.txt.
2. Build the request body with max_tokens set to 1 so you only look at the input count:

jq -n --arg t "$(cat probe.txt)" --arg m "<model id>" \
  '{model:$m, max_tokens:1, messages:[{role:"user",content:$t}]}' > req.json

ask https://official-address $OFFICIAL_KEY req.json | jq '.usage.prompt_tokens'
ask https://relay-address $RELAY_KEY req.json | jq '.usage.prompt_tokens'

3. The two numbers should be identical or differ by only a few tokens (fixed message-template overhead). If the relay returns tens or hundreds more, it has usually injected a system prompt. If the gap is proportional (say 20% more), the tokenizer is probably not from the same vendor at all.

Anthropic also offers a token-counting endpoint that doesn't consume quota (count_tokens in the Messages API), and for OpenAI-family models you can count locally with the official open-source tiktoken. Note that the usage field itself can also be rewritten, so read this check together with the ones below.

#Check 2: Where is the context length boundary?

Every model has a published context limit. If the relay swapped in a model with a smaller limit, or truncates long conversations in the middle, a boundary test shows it directly.

1. Look up the model's context limit in the official docs.
2. Build an input near the limit (for example 90% of it), put a random string at the very beginning, and ask at the end "what was the string at the beginning?"
3. Then build an input that clearly exceeds the limit.

How to read it: the official API returns an explicit error when the limit is exceeded. If the relay answers normally to an over-limit request, it is silently truncating content. If the 90% request can't recall the opening string, the usable context is smaller than advertised.

#Check 3: Do tool-call format and behavior match?

Tool calling (function calling) differs in the details across vendors: how the argument JSON is written, how many calls come back at once for parallel calls, the format of the tool_call id, and the values of stop_reason or finish_reason. These are protocol-level details, and a model swap tends to expose them.

Steps: define two simple tools (for example weather lookup and time lookup), send a question that needs both, run it 5 times each, and record:

• Whether both calls come back at once or over two rounds
• The prefix and length of the call id
• The value of the finish reason field
• Any extra fields or format errors in the arguments

Compare the official and relay results side by side. Differences in structure say more than differences in content.

#Check 4: Do latency and throughput look similar?

Different models and deployments have distinctive generation speeds. Use streaming output to measure two numbers: time to first token (TTFT) and output tokens per second.

# Record time to first byte and total time; set BASE and KEY to the official or relay address and key
jq '. + {stream:true, max_tokens:500, stream_options:{include_usage:true}}' req.json > stream.json
curl -s -N "$BASE/v1/chat/completions" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d @stream.json -o out.txt \
  -w "time to first byte %{time_starttransfer}s total %{time_total}s\n"
# usage.completion_tokens in the last data chunk is the output token count
grep -o '"completion_tokens":[0-9]*' out.txt | tail -1

Throughput is roughly output tokens ÷ (total time − time to first byte). Some services send response headers before generation starts, so time to first byte is only an approximation of TTFT. stream_options is an OpenAI-compatible parameter that some relays may not support; if so, drop it and use the usage from a non-streaming request to count output tokens.

TTFT is affected by the network. An extra hop through a relay adding a few hundred milliseconds is normal, so you can't conclude anything from that alone. For how to separate connection setup, gateway round trip, and time to first token, see Breaking down AI API latency: connection setup, gateway round trip, and time to first token. Throughput is more discriminating: the network only affects the start, not the speed during generation. If the relay's tokens per second are much higher than the official API's, suspect a smaller model.

For the network part, you can first use cdnping's online speed test to see first-connection and round-trip times from your location to each vendor's official API entry point, which separates network factors from model factors.

#Check 5: Is the output distribution for fixed prompts consistent?

A single answer isn't comparable, but the distribution over many answers is. Pick a few questions that have a definite answer and some difficulty (for example a multi-step arithmetic problem, or a code question with a fixed output), set temperature to 0, and run each 10 times:

• Count the accuracy
• Look at the range of output lengths
• Look at phrasing habits, such as whether it tends to use lists or opens with a fixed sentence pattern

At temperature 0, the same model usually produces highly similar output (not necessarily identical). If the relay's accuracy is clearly lower, or the style is completely different, read it together with the earlier checks. This check is heavily affected by quantization, so small differences on an open-weight model are not necessarily a swap.

#How do you interpret the results?

In practice, the first three checks are the most reliable, and the last two serve as corroboration. If token counts and tool calls both match, you can basically rule out a model swap; what remains is most likely quantization, parameters, or the network.

#What should you keep watching after the checks?

One check only describes the present. A relay's upstream can change at any time, and the official model itself gets updated. A few low-cost habits:

• Save the scripts above and run them weekly, or whenever the model feels dumber, then compare with your first results.
• Watch quantization precision and availability of open-weight models across inference providers at /en/models. In the same 2026-10-17 snapshot, across the 36 endpoints of that lightweight DeepSeek model, the lowest 24-hour availability was about 88.1%, 4 endpoints were below 90%, and 22 were at 100%. For a detailed analysis, see How much can uptime differ between OpenRouter providers for the same model?
• Track official changelogs and pricing pages at /en/changes. If the vendor changed the default version, the difference you see may come from upstream rather than the relay.
• When a model suddenly feels dumber, first check /en/down for other people's reports of degraded quality or outages, then decide whether your own channel is the problem.

Asking the right questions before choosing a relay is easier than detecting problems afterward: who the upstream is, whether a system prompt is injected, whether responses are cached, and how usage is metered. For the full checklist, see How to choose an AI API relay: 8 checks on availability, billing, and data policy. For Claude-specific risks, see Is a Claude API relay reliable? Main risks and how to self-check.

One last reminder: whatever third-party channel you use to access a model, follow the upstream service's terms of use and local regulations. These checks are only for verifying that the service you bought matches what was advertised.