Blog / Performance

AI API Latency Breakdown: Connect Time, Gateway RTT, and TTFT

2026-10-17cdnping editors8 min read

The latency of an AI API call is usually three delays stacked together: establishing the connection (DNS, TCP, TLS), one round trip to the gateway, and the queueing and prefill that happen before the model emits its first token. The first two are mostly determined by the network. The third is mostly determined by the server and the request itself. With curl -w and one streaming request, you can measure all three separately, then decide whether to fix the network, the code, or the request.

#What the three segments are

#Segment 1: connection setup

The first time a client reaches an API domain, it has to do three things:

• DNS resolution: turn api.example.com into an IP address. With a local cache hit this is close to 0. Without one, it depends on how far away the resolver is and on the recursive lookup chain.
• TCP handshake: 1 round trip (RTT).
• TLS handshake: 1 RTT for TLS 1.3, 2 RTTs for TLS 1.2.

So a brand-new connection costs at least 2 to 3 RTTs before any application data moves, plus DNS time. If the RTT from the client to the API entry point is 200 ms and you use TLS 1.3, connection setup alone takes about 400 ms plus DNS. The larger the RTT, the more this segment is amplified.

#Segment 2: gateway round trip

Once the connection is up, sending a request and receiving the response headers takes one network RTT plus the processing time at the server entry point (load balancing, authentication, rate limiting, routing). No model inference is involved, so you can measure this segment on its own with a request that does not need a model, for example calling /v1/models without a key. The gateway normally answers with a 401 right away. What you measure is "network + entry-point processing."

If there is a proxy, relay, or aggregation gateway in the path, every extra hop adds another round trip and another round of processing to this segment.

#Segment 3: time to first token (TTFT)

TTFT (Time To First Token) is the time from sending the request to receiving the first piece of generated content. If the connection is new, it includes the first two segments, plus the following on the server side:

• queueing for inference capacity;
• prefill, where the model processes the whole input prompt (the longer the input, the longer this takes);
• on endpoints with reasoning or thinking mode enabled, the model first generates thinking content that is either hidden or returned separately, so the first token of the actual answer arrives later.

After TTFT comes generation speed (tokens output per second), which determines how long the whole answer takes. It is outside the three segments discussed here.

#Summary table

#Measuring each segment with curl

The commands below run on macOS and Linux. On Windows, run them in WSL or Git Bash. Replace API_HOST with the entry point you want to test.

#Step 1: measure each phase of connection setup

API_HOST=api.openai.com

curl -s -o /dev/null https://$API_HOST/v1/models -w '
DNS lookup          : %{time_namelookup}s
TCP connected       : %{time_connect}s
TLS handshake done  : %{time_appconnect}s
Before request sent : %{time_pretransfer}s
First byte received : %{time_starttransfer}s
Total time          : %{time_total}s
HTTP status         : %{http_code}
'

Every value curl -w prints is cumulative from the start of the timer, so you need to subtract to get each phase:

• DNS = time_namelookup
• TCP handshake = time_connect − time_namelookup, roughly 1 RTT
• TLS handshake = time_appconnect − time_connect
• Gateway round trip = time_starttransfer − time_pretransfer

This request carries no key, so it should return 401. The gateway issues the 401 without involving a model, which means time_starttransfer − time_pretransfer corresponds to Segment 2.

#Step 2: see how much connection reuse saves

If you list the same URL several times in one curl command, curl reuses the first connection:

curl -s -o /dev/null -o /dev/null -o /dev/null \
  https://$API_HOST/v1/models \
  https://$API_HOST/v1/models \
  https://$API_HOST/v1/models \
  -w 'connect=%{time_connect}s tls=%{time_appconnect}s total=%{time_total}s new_conns=%{num_connects}\n'

On the first line, connect and tls have real values. On the next two lines they are close to 0, and new_conns of 0 means no new connection was opened. The difference between the first line's total and the other two is roughly what connection reuse saves on each request. The "round trip" metric on the cdnping home page works the same way: it reuses the connection, sends 3 requests, and takes the median.

#Step 3: measure real TTFT

time_starttransfer cannot be used directly as TTFT. Streaming endpoints often send the response headers first and push content later, so the first byte can arrive before the first token. The script below uses the first data: line that carries content:

API_HOST=api.openai.com
API_KEY=your-api-key
MODEL=your-model-name

start=$(date +%s%N)
curl -sN https://$API_HOST/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$MODEL\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Say one word\"}]}" \
| while IFS= read -r line; do
    case "$line" in
      *'"content":"'[!\"]*)
        now=$(date +%s%N)
        echo "TTFT: $(( (now - start) / 1000000 )) ms"
        break ;;
    esac
  done

The date that ships with macOS does not support %N. Run brew install coreutils first, then replace date with gdate. Anthropic-format endpoints use a different path, different headers, and a different event structure, so adjust the matching condition according to the official documentation.

#Step 4: put the three segments side by side

Run each measurement 3 to 5 times at the same time and on the same network, take the median, and fill in this table:

Look at which line dominates. If the TCP handshake alone takes several hundred milliseconds, check the network path first. If connection setup and the round trip are both low but TTFT is long, the problem is on the server or in the request itself. Shorten the prompt, switch to a lighter model, and measure again.

#What connection reuse and streaming each shorten

#Connection reuse: removes Segment 1

The most common mistake in code is creating a new HTTP client for every request. The official SDKs keep a connection pool inside the client instance, but only if you reuse the same instance:

# Not recommended: opens a new connection on every call
def ask(q):
    client = OpenAI()
    return client.chat.completions.create(...)

# Recommended: reuse one client for the whole process
client = OpenAI()
def ask(q):
    return client.chat.completions.create(...)

A few more points:

• Servers close connections that sit idle for a while. A low-traffic service can send a lightweight request before the real one to warm the connection.
• HTTP/2 allows many concurrent requests on a single connection. Under heavy concurrency it saves handshakes compared with opening many HTTP/1.1 connections.
• When a serverless function cold-starts, its connection pool is empty, so the first call always pays the connection setup cost.

#Streaming: TTFT stays about the same, but users see content sooner

Setting stream: true does not make the model produce its first token any faster. What it changes is when the user sees content: a non-streaming call returns only after the entire answer is generated, while a streaming call starts displaying as soon as the first token exists. The longer the answer, the bigger the difference in perceived speed.

Streaming has another use. The arrival time of the first content event is the most accurate TTFT you can get, which makes it handy for monitoring. If there is a reverse proxy in the path, make sure it does not buffer SSE responses. Otherwise streaming degrades into a single all-at-once response.

#How to read cdnping edge probe data

The cdnping API status page probes each official API entry point from the Cloudflare edge every 5 minutes. The table below comes from a cdnping snapshot taken on October 17, 2026. The edge node running the probes was DEN (Denver, USA), and the window ran from 16:40 to 18:05 Beijing time (UTC+8), about 85 minutes, with 242 probes across all entry points. cdnping only began collecting on 2026-10-17, so this data covers a short period and is not a long-term statistic. The data is published under CC BY 4.0. For the measurement method, see the methodology page.

Three things to keep in mind when reading this table:

1. p50 is the median: half of the probes were faster than this. p95 means 95% of probes were no slower than this value, so it reflects jitter. For example, api.openai.com has a p50 of 96 ms and a p95 of 194 ms: stable most of the time, with occasional readings of about double.
2. These measurements were taken from Denver, USA. Overseas entry points sit between a few dozen and a little over a hundred milliseconds, and the China-based entry points (Moonshot AI's Kimi, Zhipu, Volcengine Ark, Alibaba Cloud Model Studio's DashScope, SiliconFlow, DeepSeek) sit at 350 to 630 ms. This mainly reflects the network distance from Denver to each entry point, not how fast any model is. Measured from mainland China, the ordering would very likely be completely different, so measure on your own network to get real numbers.
3. The probes do not trigger model inference. They correspond to Segments 1 and 2 of this article and exclude the model-side portion of TTFT.

An aggregation gateway adds a hop. Take AGI Plan as an example (AGI Plan is a commercial partner of cdnping): in the same snapshot, agiplan.dev had a p50 and p95 that were both 883 ms. Identical values mean this window cannot show anything about jitter, so don't draw conclusions from it. A request through a gateway travels two network legs, client to gateway and gateway to upstream. When evaluating this kind of service, run Steps 1 and 3 above on your own network and compare connection setup, round trip, and TTFT separately.

#What to check next

• Open the cdnping live speed test to see "first connection" and "round trip" for 15 AI API entry points measured from your current network in the browser, and compare them with your curl results.
• If the round trip is noticeably higher than expected, use the IP and environment check to look at your exit IP's location and latency profile. It compares the browser round trip with the exit TCP round trip measured by Cloudflare, which helps you tell whether your requests are passing through a relay. The results can be turned into a watermarked image, and you can choose to hide the IP address before sharing.
• For a single service, look at a per-service page such as OpenAI API status to tell whether the slowness is on your side or the entry point itself is jittering.
• To understand overall network conditions when reaching overseas AI APIs from mainland China, read AI API latency from China.

Some overseas AI services have regional restrictions. Follow the provider's terms of service and comply with local laws and regulations.