--- title: "API Uptime Testing: Measure AI API Uptime Yourself with a Script" description: "A ready-to-run AI API probe script, plus how synthetic probes differ from real-request stats, how to count 4xx vs 5xx, and how many samples you need." url: https://cdnping.org/en/blog/measure-ai-api-uptime-yourself language: en published: 2026-10-18 category: "Performance" tags: ["API uptime","monitoring script","relay services","uptime calculation","AI API"] source: cdnping.org license: Free to quote and summarize with a link to the url above. --- # API Uptime Testing: Measure AI API Uptime Yourself with a Script The simplest reliable way to measure API uptime is to send a minimal request to the endpoint at a fixed interval, record the status code and latency, and compute success / (success + server-side failures). The hard part is not the script. It is the definitions: which errors count against the service and which are yours, whether you have enough samples, and whether you are testing the front door or real inference. This article gives you a script you can run as-is, then uses two public datasets from cdnping and AGI Plan to show how the same service can look very different depending on how you count. ## Are synthetic probes and real-request stats measuring the same thing? No. There are two common approaches: - Synthetic probes: the monitoring side sends requests on a schedule, for example one request to the API endpoint every 5 minutes. The intervals are even and you get data at 3 a.m. when nobody is using the service. The downside is that the request is light. It often only proves the gateway responds, not that it reached model inference. - Real-request stats: the service operator aggregates the success and failure counts of actual user requests. This reflects what users really experienced, but volume follows the daily rhythm. When an hour has only two or three requests overnight, the ratio is statistically meaningless. The cdnping [API status page](/en/status) is the first kind: it probes each official API endpoint from the Cloudflare edge every 5 minutes, and the details are in the [methodology](/en/methodology). The hourly data published by AGI Plan is the second kind. AGI Plan is a commercial partner of cdnping, which makes it convenient to put the two approaches side by side. ## How big is the difference between the two approaches? Start with synthetic probing. This is a cdnping snapshot from October 17, 2026 at 18:05 (Beijing time, UTC+8), probed from Denver (DEN). cdnping only began collecting data that day, so although the field is named "24 hours", it actually covers about 85 minutes (16:40 to 18:05) and 242 probe records: | Endpoint | Median latency (p50) | p95 | Availability in window | |---|---|---|---| | api.anthropic.com | 73 ms | 84 ms | 100% | | api.openai.com | 96 ms | 194 ms | 100% | | api.deepseek.com | 353 ms | 414 ms | 100% | | api.moonshot.cn | 375 ms | 1125 ms | about 94.4% | | agiplan.dev | 883 ms | 883 ms | 100% | Of the 15 endpoints, only the Kimi (Moonshot AI) open platform had any failures in this period. All the others were at 100%. Now real requests. AGI Plan publishes hourly counts. Taking the 48 hours from 19:00 on October 15 to 18:00 on October 17 (Beijing time), there were 13,963 successes and 2,177 platform failures, which works out to about 86.5% by the formula. The failures are highly concentrated: the hours at 14:00 and 15:00 on October 16 had 722 and 877 failures respectively, for an hourly success rate of about 48% and 34%. From 4 a.m. to 6 a.m. there were only 2 requests per hour, so a 100% success rate there tells you nothing. In the same snapshot, AGI Plan's 30-day availability by capability group also differs: the fast group is about 81.7% (44,729 requests), the frontier-model group about 94.4%, the balanced group about 95.0%, and the China-made models group about 98.0%. Putting the two together: during the synthetic probe window, the agiplan.dev endpoint was 100% available, yet in the same period (16:00 to 18:00 on October 17) real requests had 1 to 7 failures every hour. These do not contradict each other. A responding endpoint only shows that the gateway is online. When an upstream model returns 5xx, an endpoint probe cannot see it. Conversely, real-request data alone leaves you with nothing to judge by during quiet hours. So when you measure on your own, do both, or at least be clear about which kind of number you are holding. ## Which errors should count as downtime? How you define the denominator determines the result. Here is a classification table you can adopt directly: | Result | Examples | Counts toward uptime? | Notes | |---|---|---|---| | 2xx | 200 | Count as success | For streaming endpoints, confirm you received the end-of-stream event | | 5xx | 500, 502, 503, 504 | Count as failure | Problem on the provider or gateway side | | Timeout, connection failure | DNS failure, TLS handshake failure, read timeout | Record separately, then decide | Could be the service, could be your network | | 429 | Rate limiting | Track separately | Tied to your quota and concurrency, so don't treat it as an outage outright | | Other 4xx | 400, 401, 403, 404 | Excluded | Parameter, key, or permission problems; the caller's fault | AGI Plan's public data uses the "only 5xx counts as a platform failure" definition. When you test yourself, we also recommend keeping 4xx out of the denominator, but always log them. A sudden wave of 401s may mean a key was rotated, and a 403 may relate to region or account status. Those should be handled under the provider's terms of service and are not uptime issues. Timeouts are the hardest to attribute. A practical approach is to probe a control target at the same time, such as cdnping's own endpoint. If the control also times out, it is most likely your local network. If the control is fine and only the AI endpoint times out, then count it against the service. For the order in which to check the network layer, see [How to troubleshoot Claude Code connection timeouts and errors](/en/blog/claude-code-connection-troubleshooting). ## Hands-on: a minimal probe script The Python script below uses only the standard library and works with any OpenAI-compatible endpoint (official APIs, cloud-hosted offerings, or aggregation gateways). Each run sends two requests: a `GET /models` to check the endpoint, and a chat request with `max_tokens=1` to check the inference path, and appends the results to a CSV. ```python #!/usr/bin/env python3 # probe.py: probe once per run, append results to probe.csv import csv, json, os, time, urllib.request, urllib.error BASE = os.environ["API_BASE"].rstrip("/") # e.g. https://api.example.com/v1 KEY = os.environ["API_KEY"] MODEL = os.environ["API_MODEL"] # the model id you actually use OUT = os.environ.get("PROBE_OUT", "probe.csv") def call(kind, method, path, body=None): req = urllib.request.Request(BASE + path, method=method, data=json.dumps(body).encode() if body else None, headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) t0 = time.time(); status, err = 0, "" try: with urllib.request.urlopen(req, timeout=30) as r: r.read(); status = r.status except urllib.error.HTTPError as e: status = e.code except Exception as e: # timeout, DNS, TLS, etc. err = type(e).__name__ ms = int((time.time() - t0) * 1000) return [int(t0), kind, status, ms, err] rows = [ call("entry", "GET", "/models"), call("infer", "POST", "/chat/completions", {"model": MODEL, "max_tokens": 1, "messages": [{"role": "user", "content": "ping"}]}), ] with open(OUT, "a", newline="") as f: csv.writer(f).writerows(rows) ``` Run it every 5 minutes with cron: ```bash # crontab -e */5 * * * * API_BASE=https://api.example.com/v1 API_KEY=sk-xxx API_MODEL=your-model PROBE_OUT=$HOME/probe.csv /usr/bin/python3 $HOME/probe.py ``` The inference request consumes a small number of tokens. At one request every 5 minutes with 1 output token each, that is 288 per day, and the cost is negligible, but we still recommend a dedicated key so you can tell it apart on the bill. For how endpoint URLs are written for different services, see [Custom API endpoint configuration for Claude Code, Codex, and OpenCode](/en/blog/configure-custom-api-endpoint-claude-code-codex). When summarizing, compute using the definitions above: 4xx stays out of the denominator, and 429s and timeouts are listed separately: ```bash awk -F, '$2=="infer"{ if($3>=200 && $3<300) ok++; else if($3>=500) fail++; else if($3==429) rl++; else if($3==0) to++; else c4++ } END{ if (ok+fail==0) { print "no countable samples"; exit } printf "success %d 5xx %d uptime %.3f%%\n", ok, fail, 100*ok/(ok+fail); printf "429 %d timeout/connection failure %d other 4xx %d\n", rl, to, c4 }' ~/probe.csv ``` If you later confirm that a timeout was the provider's fault, add it to `fail` and recompute. Keep both numbers. ## How long do you need to test to get enough samples? You can estimate it with simple arithmetic. One probe every 5 minutes gives 288 samples a day, and a single failure drops that day's uptime by 0.35 percentage points. To tell 99.9% from 99.5%, you need at least a thousand samples: out of 1,000 samples, the former corresponds to about 1 failure and the latter to about 5, and with fewer samples you cannot separate them. At one probe every 5 minutes, that means three to four days at minimum; a monthly view needs about 8,640 samples, where a single failure is about 0.01%. A few rules of thumb: 1. A single day can only show whether there is a big problem. Don't use it to decide which of two services is more stable. 2. Charting by the hour is more useful than looking at one overall percentage. In the AGI Plan 48-hour data above, the 86.5% total hides the fact that the failures were concentrated in two hours. 3. Don't draw conclusions from hours with very few requests. Set a threshold, for example gray out any hour with fewer than 10 samples. 4. Probe from a network environment similar to your users'. Results measured from an overseas data center do not represent what a mainland China office network sees. For the causes of latency differences, see [Breaking down AI API latency](/en/blog/api-latency-connect-rtt-ttft). ## How do you compare public data with your own? The biggest problem with self-testing is not knowing whether a failure is your network or the provider. You can cross-check against several public sources: - cdnping [/status](/en/status) and [/down](/en/down): edge probes, official status page sync, and user reports in one place, useful for judging whether there is a widespread outage. If you only suspect your own network, start with [Is ChatGPT down? Three steps to tell a service outage from a network problem](/en/blog/is-chatgpt-down-or-my-network). - Official status pages: for how to read incident severity and affected components, see [How to read the OpenAI and Claude status pages](/en/blog/how-to-read-ai-status-pages). - Real-request data published by providers: AGI Plan, for example, lets you download hourly counts per capability group without logging in, as CSV by default: ```bash curl -s 'https://agiplan.dev/api/status/requests?month=2026-10' -o agiplan-2026-10.csv curl -s 'https://agiplan.dev/api/status/requests?month=2026-10&format=json' ``` The cdnping [/status/agiplan](/en/status/agiplan) page mirrors this data. Once you have it, align it by hour with your own `probe.csv`: if failures show up on both sides at the same time, you can generally attribute them to the provider; if only your side fails, check your local network and egress first. If what you are testing is a relay or aggregation gateway (a third-party service that resells or routes access to model APIs), uptime is only one factor. Billing, data policy, and model consistency also need checking; see [How to choose an AI API relay: 8 checks on uptime, billing, and data policy](/en/blog/api-relay-how-to-evaluate) for the full list. When traffic goes through a router such as OpenRouter, the same model can have different uptime across providers; see [Same model, different OpenRouter provider: how much can uptime differ](/en/blog/same-model-different-providers-uptime). ## Pre-launch checklist - [ ] Two kinds of probes: endpoint (`/models`) and inference (minimal chat request) - [ ] 2xx counts as success, 5xx as failure; 429, timeouts, and other 4xx are logged separately - [ ] A control target is probed at the same time to separate local network problems - [ ] The probe location is in a network environment similar to real users - [ ] At least 3 to 4 days of data before comparing, and the window and sample count stated in any report - [ ] Summarized by the hour, with low-sample hours not used for conclusions - [ ] A dedicated key, compliance with the terms of service, and no using probes to test rate limits or region restrictions Whenever you quote an uptime number, write the source, window, and definition next to it. The cdnping data in this article covers only about 85 minutes, and the AGI Plan hourly data covers 48 hours. They illustrate differences in method, not long-term conclusions.