---
title: "AI App API Monitoring: From Upstream Probes to Alerts, a Checklist"
description: "Monitor the LLM APIs your app relies on in three layers: reachability probes, real-request success and latency, and output checks. Includes a curl script."
url: https://cdnping.org/en/blog/ai-app-availability-monitoring-guide
language: en
published: 2026-10-22
category: "Performance"
tags: ["API monitoring","availability","LLM API","alerting","observability"]
source: cdnping.org
license: Free to quote and summarize with a link to the url above.
---
# AI App API Monitoring: From Upstream Probes to Alerts, a Checklist

Monitor an AI app's APIs in three layers. First, probe the upstream model endpoints with requests that cost no tokens, to see whether you can connect at all. Second, record the success rate and latency of every real call inside your app. Third, spot-check output quality, because a 200 response does not mean the answer is right. If you only do the first layer, you will miss rate limiting, timeouts, and "the model got dumber." If you only watch the vendor's status page, you will learn about outages later than your own users do. Below are probe frequencies, probe locations, the fields to record, and a checklist you can follow as written.

## Why isn't the official status page enough?

An official status page is a set of incident announcements that the vendor publishes by hand. It has two built-in limits. It lags, because engineers have to confirm the blast radius before they post an incident. And it describes component status from the vendor's point of view, with no knowledge of where your requests start or which route they take.

You also need to know how to read the incident severity levels. For example, in the status-page records cdnping has synced since 2026-09-17, Cloudflare has 19 minor and 3 major incidents, and OpenAI has 19 minor and 4 major. Some affect only one region or sub-product: Cloudflare's two minor incidents on 2026-10-16 were increased 5xx errors in Ashburn and network performance problems in Thailand. Whether a given incident matters to you depends on which endpoints you actually call. For how to read them in detail, see [How to read the OpenAI and Claude status pages](/en/blog/how-to-read-ai-status-pages).

So a status page works as evidence that "this is the vendor's problem." It should not be your first alert source. Your first alert should come from your own probes and instrumentation.

## Layer 1: How do you probe external reachability?

This layer answers one question: from a given location, can a request reach the API entry point and get a response? It ignores model output and checks only whether DNS, TCP, TLS, and HTTP are healthy.

### What request can you probe with at no cost?

Don't use real chat requests for high-frequency probing. They burn tokens, and they mix model inference variance into your network metrics. Two common approaches:

- Request a path that requires authentication, such as `GET /v1/models`, without an API key. A 401 means DNS, TLS, and the gateway all work and only authentication failed, which is exactly what a reachability probe wants to see.
- Request the model-list endpoint with a key. These endpoints are generally not billed per token (check each vendor's official docs for billing), and they also confirm that your key is valid.

Here is a probe script you can run as is. It uses curl's timing variables to split the request into phases:

```bash
#!/usr/bin/env bash
# probe.sh: run one reachability probe against several API endpoints, one record per line
endpoints=(
  "openai https://api.openai.com/v1/models"
  "anthropic https://api.anthropic.com/v1/models"
  "deepseek https://api.deepseek.com/models"
)
for e in "${endpoints[@]}"; do
  set -- $e
  curl -s -o /dev/null --max-time 10 \
    -w "$(date -u +%FT%TZ) $1 code=%{http_code} dns=%{time_namelookup} tcp=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
    "$2" || echo "$(date -u +%FT%TZ) $1 code=000 error=curl_exit_$?"
done
```

The classification rules can be simple. A `code` of 2xx, 401, or 403 counts as reachable. `000` (connection failure or timeout) and 5xx count as unreachable. Record 429 separately as rate limited, and don't fold it into availability. Paths follow each vendor's official docs; the ones above are examples.

### How often should you probe?

Start at once every 5 minutes. cdnping's edge probes use the same interval. Probe too often and a single network blip generates a flood of alerts. Probe too rarely, say every 15 minutes, and a 20-minute outage may be caught by only one or two samples. Write the alert condition as "2 consecutive failures" or "2 failures in the last 3 probes," not "alert on any single failure."

### Where you probe from changes the result

If your app calls both overseas APIs and Chinese APIs, the probe location can change the measured latency, and even availability, a lot.

Take cdnping's [API status page](/en/status) as an example. It probes each vendor's official API endpoint from the Cloudflare edge every 5 minutes. The table below is a snapshot of the window 2026-10-17 08:40-10:05 (UTC), from a probe node in Denver (DEN), with 242 probe records in total. cdnping only started collecting on 2026-10-17, so the "24h" in the field names really covers just this hour or so. Treat the table as an illustration of the method, not a long-term conclusion.

| Service | Endpoint | p50 (ms) | p95 (ms) | Availability in window |
|---|---|---|---|---|
| OpenAI API | api.openai.com | 96 | 194 | 100% |
| Claude API | api.anthropic.com | 73 | 84 | 100% |
| Gemini API | generativelanguage.googleapis.com | 69 | 89 | 100% |
| DeepSeek API | api.deepseek.com | 353 | 414 | 100% |
| Kimi Open Platform (Moonshot AI) | api.moonshot.cn | 375 | 1125 | 94.4% |
| Alibaba Cloud Model Studio | dashscope.aliyuncs.com | 615 | 1152 | 100% |
| Volcengine Ark | ark.cn-beijing.volces.com | 485 | 1149 | 100% |

Source: cdnping edge probes, open data at `/api/status`, CC BY 4.0. Definitions are in the [methodology](/en/methodology).

Two things stand out in this table:

1. From a US node, the overseas APIs show a p50 under 100 ms, while the Chinese APIs show 350-615 ms. This mainly reflects the distance from the probe to the endpoint, not that the Chinese APIs are slow. Run the same script from a data center in China and the gap would likely shrink or even reverse. Judging "which vendor is faster" from a single location is a mistake.
2. In a short window, one failure visibly drags availability down. The Kimi Open Platform shows 94.4% here. At one probe every 5 minutes, a little over an hour gives about 18 samples, and one failure is 17/18 ≈ 94.4%. When you read an availability figure, read the sample size with it.

In a real deployment, put probes in the same network environment as your users and your servers where you can:

| Probe location | Question it answers | What to put there |
|---|---|---|
| The data center where your app servers run | Can my backend reach the upstream? | Required. This is the main alert source. |
| Another data center or cloud region in the same country | Is it a single-site routing problem or a general one? | Use it to rule out network faults at your own site. |
| An overseas node or edge network | Is the upstream endpoint itself healthy? | Start with public data such as cdnping /status as a reference. |

Before you build your own, you can use [/status](/en/status) and [/down](/en/down) as an external reference. If your probe fails while the public probe is healthy, the problem is most likely on your side of the network. If both fail, then go check the status page. The reasoning is the same as in [Is ChatGPT down? Three steps to tell a service outage from your own network problem](/en/blog/is-chatgpt-down-or-my-network).

## Layer 2: How do you instrument success rate and latency on real requests?

A passing reachability probe doesn't mean user requests succeed. Rate limits, exhausted quota, long-context timeouts, and streaming responses that drop midway only show up on real requests. So your application should write a record for every model call.

### Which fields should you record on every call?

| Field | Description | Use |
|---|---|---|
| ts | Request start time | Time series |
| provider / endpoint | The vendor and endpoint called | Split by upstream |
| model | The model name in the request and the one in the response | Catch silent model swaps |
| status | HTTP status code; for streaming requests, also whether the stream ended cleanly | Compute success rate |
| error_type | Timeout, connection failure, 429, 5xx, content filter, etc. | Assign responsibility |
| connect_ms | Time to establish the connection | Separate network from inference |
| ttft_ms | Time to first token | Latency as users feel it |
| total_ms | Total time | Investigate long requests |
| input_tokens / output_tokens | Usage | Reconciliation, throughput |
| retry_count | Number of retries | Keep retries from hiding failures |
| request_id | The request ID returned by the upstream | Support tickets with the vendor |

Split latency into at least three parts: connection, time to first token, and total time. The reasons and the method are covered in [Breaking down AI API latency: connection setup, gateway round trip, and time to first token](/en/blog/api-latency-connect-rtt-ttft). If you record only total time, the numbers stop being comparable as soon as output length changes.

### How should you define availability?

Without a shared definition, different teams produce numbers that can't be compared. Write the definition down:

- Numerator: the number of successful requests.
- Denominator: successes plus requests that failed because of the upstream itself.
- 5xx, connection failures, and timeouts count as upstream failures. A 400 for bad parameters or a 401 for a bad key is your own problem and stays out of the denominator. Count 429 separately.
- If a request succeeds after a retry, the first failure still counts as one failure. Otherwise retries hide the outage.

You can find a similar definition in public data. For example, AGI Plan's public hourly request stats (`https://agiplan.dev/api/status/requests?month=YYYY-MM`) count only 5xx as platform failures, so availability = successes / (successes + platform failures). AGI Plan is a commercial partner of cdnping; I mention it here only as a reference for the definition. For a full script and a longer discussion of how to compute availability yourself, see [Measuring AI API uptime yourself: metrics, scripts, and definitions](/en/blog/measure-ai-api-uptime-yourself).

### How do you set alert thresholds?

Run for a week or two and set thresholds from your own baseline. The ones below are only starting examples:

- In a 5-minute window, an upstream's failure rate is several times its usual baseline, and there are at least 20 requests.
- p95 time to first token stays above twice the baseline for 15 minutes in a row.
- A sudden rise in the share of 429s usually points to a quota or rate-limit problem. Notify whoever owns the account and quota, not the on-call network engineer.

In low-traffic hours such as late at night, set a minimum sample size on the threshold. Otherwise 1 failure out of 3 requests will fire an alert.

## Layer 3: How do you catch a 200 response with a wrong answer?

Ordinary API monitoring usually doesn't need this layer. The upstream may have switched inference providers or quantization precision. The API returns 200 the whole time, and users only feel that the model "got dumber."

What you can do:

1. Keep a fixed set of regression samples. For example, prepare 20 to 50 prompts with definite answers, such as fixed-format JSON extraction, simple arithmetic, or classification on specified keywords. Run them daily or hourly and record the pass rate.
2. Check the model name in the response. If the requested model and the returned model differ, raise a separate alert. For more consistency checks, see [How to confirm which model an API is really giving you](/en/blog/verify-model-authenticity-relay).
3. Watch for provider and quantization changes. When you call open-weight models through an aggregator, the same model name can sit behind different providers at different precisions. cdnping's [model monitoring page](/en/models) is built on the public OpenRouter API. It records each inference provider's availability, latency, quantization precision, and listing changes. For how much one model differs across providers, see [Same model, different OpenRouter provider: how much can uptime differ?](/en/blog/same-model-different-providers-uptime). For background, see [Did the model get dumber, or did the provider change? Inference providers and fp8/fp4 quantization](/en/blog/llm-quantization-fp8-fp4-providers).
4. Treat user feedback as a signal. A sudden rise in thumbs-down or regenerate rates can show a problem earlier than regression samples do.

Regression samples consume tokens, so you don't need to run them often. But you must fix the prompts and parameters (temperature at 0 or as low as possible). Otherwise most of the variation in results comes from sampling randomness.

## A checklist you can follow as written

### Reachability probes

- [ ] Every upstream endpoint has a no-token probe, every 5 minutes
- [ ] Probes cover at least the data center where your app servers run, ideally plus one location on a different network
- [ ] 401/403 count as reachable, 000 and 5xx as unreachable, 429 is listed separately
- [ ] Alerts fire on several consecutive failures, not a single failure
- [ ] You have an external reference (for example [/status](/en/status)) to tell your own network problems from upstream problems

### Real-request instrumentation

- [ ] Every call records the fields in the table above, at least status code, error type, time to first token, and request_id
- [ ] The availability definition is written down: which errors count as upstream, which as yours, and how retries are counted
- [ ] Charts are split by vendor, model, and endpoint, not just a global success rate
- [ ] Alert thresholds are based on your own baseline and have a minimum sample size

### Output quality

- [ ] A fixed set of regression samples runs on a schedule and records the pass rate
- [ ] The model name in the response is checked
- [ ] You watch the upstream's providers, quantization precision, and website updates (you can subscribe to the RSS feed at [/changes](/en/changes))

### During an incident

- [ ] Look at your own probes and instrumentation first, then public probes, and the official status page last
- [ ] Keep the request_id of failed requests so you can report them to the vendor
- [ ] If you have a backup upstream, write the switch-over and switch-back conditions in advance

## Which one to do first

The three layers differ a lot in cost. Reachability probes cost almost nothing and can be set up in half a day. Instrumentation means changing application code. Quality spot checks need samples that someone keeps maintaining. If you can only do one thing first, run the probe script on your application server, then compare it against the public data at [/status](/en/status). When the upstream has a problem, this lets you find out before user complaints arrive, and it tells you whether the fault is in your own route or in the upstream.

When you call AI services across borders, follow each service's terms of use and the laws where you are. Which regions a service supports is defined by its official documentation.