---
title: "Synthetic Monitoring vs Real User Monitoring for AI Apps"
description: "Synthetic monitoring vs real user monitoring (RUM) for LLM apps: a comparison table, curl probe, browser RUM snippet, and a minimal rollout plan."
url: https://cdnping.org/en/blog/synthetic-monitoring-vs-real-user-monitoring-ai
language: en
published: 2026-10-25
category: "Performance"
tags: ["monitoring","synthetic monitoring","RUM","availability","AI API"]
source: cdnping.org
license: Free to quote and summarize with a link to the url above.
---
# Synthetic Monitoring vs Real User Monitoring for AI Apps

The difference between synthetic monitoring and real user monitoring comes down to who sends the request. Synthetic monitoring means you send requests yourself, at a fixed interval from fixed locations, to "try it out." It answers: is the service up right now? Real user monitoring (RUM) collects data from your actual users' requests. It answers: how is it actually working for users? The first works with zero users but only represents the network the probe sits in. The second is closest to real experience but goes blank when there is no traffic.

For AI apps the gap between the two is wider. The upstream model API may be reachable while the output quality gets worse. A streaming response may take forever to deliver its first token while the HTTP status is a clean 200. Below, we apply both approaches to LLM endpoints and then show how cdnping combines them.

## What are synthetic monitoring and real user monitoring?

**Synthetic monitoring** is what Chinese teams often call "dial testing" (拨测) or "active monitoring." You write a script, for example one that hits an API endpoint every 5 minutes, and record whether it succeeded and how long it took. The requests are "synthetic": no user initiated them.

**Real user monitoring (RUM)**, also called "passive monitoring," instruments the frontend or backend and reports the status code, latency, time to first byte and so on for every real request. The volume follows your traffic: the more users, the more samples.

| Dimension | Synthetic monitoring (probes) | Real user monitoring (RUM) |
|---|---|---|
| Data source | Probe requests you send | Real user requests |
| When there is data | 24/7, at a fixed interval | Only when there is traffic |
| Whose network it represents | The data center and route of the probe node | The user's own network, device and region |
| Good at finding | Upstream completely down, expired certificates, DNS failures | A slow region, a failing request type, long-tail latency |
| Blind spots | Cannot see the user's network; only tests fixed requests, so it misses "bad answers" | Few samples at off-peak hours; no data before launch |
| Baseline comparison | Yes: fixed conditions make before/after comparable | Hard: the user mix keeps changing |
| Cost | Probe frequency × number of nodes, predictable | Proportional to traffic; you need to consider sampling |

## What does each one miss in an AI app?

Most monitoring articles for ordinary web pages discuss first-paint time and metrics like LCP. AI apps add three kinds of signals, and each lands in a blind spot of one approach or the other.

### 1. The probe's network is not the user's network

A synthetic probe only tells you "it is reachable from where the probe sits." Take cdnping's [API status page](/en/status) as an example: it probes each vendor's official API endpoint from Cloudflare edge nodes every 5 minutes. In the data snapshot generated on 2026-10-17, the probes came from the DEN node (Denver), the window was 08:40 to 10:05 UTC that day, and there were 242 probes. In that window:

- The Claude API had a median latency of 73 ms and the xAI API 42 ms;
- The DeepSeek API had 353 ms, Alibaba Cloud Model Studio 615 ms, and SiliconFlow 631 ms.

These numbers only say what things look like "from an edge node in Denver." They do not mean Chinese services are slower than overseas ones. Probing from inside China usually reverses the picture. That is the limitation of synthetic monitoring: the data is accurate, but how representative it is depends on where you put the probe. To see the user's side you have to measure from the user's network, for example with the in-browser speed test on the cdnping [home page](/en), which measures "first connection" (including DNS and the TLS handshake) and "round trip" (the median of 3 requests over a reused connection). For how those two phases break down, see [AI API latency breakdown: connection setup, gateway round trip and time to first token](/en/blog/api-latency-connect-rtt-ttft).

### 2. Streaming responses: a 200 does not mean a good experience

Most LLM endpoints return results over SSE streaming. A common situation: the response headers come back quickly with status 200, but the first token arrives much later. A probe that only checks the status code reports "healthy," while the user stares at a frozen UI.

So AI app monitoring should record at least three things separately: connection setup time, time to first token (TTFT), and total response time. A synthetic probe can use a fixed small request to measure TTFT and give you a comparable baseline. RUM tells you what the TTFT distribution looks like for real long prompts and long outputs.

### 3. "The model got dumber": hard for both to measure directly

A drop in output quality is invisible at the network layer. Possible causes include a change of inference provider or a change in quantization precision (for example fp8 or fp4); for background see [Did the model get dumber, or did the provider change?](/en/blog/llm-quantization-fp8-fp4-providers) cdnping's [model watch page](/en/models) records quantization changes across providers on OpenRouter, which can serve as corroborating evidence.

Within a monitoring setup, this kind of problem is usually tackled in two ways:

- Synthetic side: prepare a few fixed prompts with deterministic answers (for example, ask the model to emit a JSON snippet in a fixed format), call them on a schedule, and check that the structure is valid and the key fields are correct. This goes one step further than checking the status code, but it only covers the questions you thought of in advance.
- User side: put an "answer was bad" feedback control in the UI and track how the feedback ratio changes.

cdnping's [Is it down?](/en/down) page follows this idea. Besides official status pages and API probes every 5 minutes, it collects one-click user reports. The report types are can't open, slow, dumber, rate limited, quota reset, and logged out. Network probes can partly catch "can't open" and "slow," but problems like "dumber," "quota reset," and "logged out" are essentially invisible to probes and only show up if users tell you.

## What does real user data look like when there is no traffic?

The blind spot of RUM is visible in a piece of public data. AGI Plan publishes availability data computed from real requests (AGI Plan is a commercial partner of cdnping). The endpoint is `https://agiplan.dev/api/status/requests?month=YYYY-MM`, which gives hourly success and failure counts, and only 5xx responses count as platform failures.

In the 48 hourly buckets from 2026-10-15 to 10-17 in the snapshot, two situations stand out:

- During an incident the signal is very clear: on the morning of October 16 (UTC), two consecutive hourly buckets show 664 successes / 722 failures and 455 successes / 877 failures. In a period like that, the real-request failure rate is hard to miss, and you would spot it without any extra probing.
- At off-peak hours there are almost no samples: the three hourly buckets from 20:00 to 22:00 UTC on October 16 have only 2 requests each. Even if the service were completely down, RUM could hardly draw a reliable conclusion; 1 failure out of 2 requests is 50%.

That is why synthetic probes need to fill in: probes at a fixed frequency still produce a steady sample at off-peak hours. Conversely, probes only cover fixed requests and cannot replace the failure rate of real requests. For how availability is defined and how error budgets are calculated, see [SLI, SLO and SLA: what is the actual difference?](/en/blog/sli-slo-sla-error-budget-explained)

## How do you choose by scenario? A decision table

| Question you want to answer | First choice | Add |
|---|---|---|
| Is the upstream model API up right now? | Synthetic probes (multiple nodes) | Official status page |
| Is the service healthy in the middle of the night when nobody is using it? | Synthetic probes | n/a |
| Are users in a certain region or ISP slower? | RUM (grouped by region) | Add a probe node in that region |
| Has first-token latency for streaming responses gotten worse? | Both: probes for the baseline, RUM for the distribution | n/a |
| Did things get worse after an SDK upgrade or gateway switch? | Synthetic probes for before/after comparison | Watch RUM during the canary |
| Has output quality dropped? | Structural checks with fixed prompts | User feedback ratio |
| Are you hitting rate limits (429)? | RUM (only real concurrency triggers them) | n/a |
| Is it a service outage or the user's own network? | Synthetic probes plus a user-side speed test for comparison | See [Three steps to tell a service outage from a network problem](/en/blog/is-chatgpt-down-or-my-network) |

## A minimal rollout plan

Here is a combination a small team can follow, in order.

### Step 1: Add one synthetic probe per upstream endpoint

The simplest version uses curl, runs every 5 minutes, and records the time spent in each phase:

```bash
curl -o /dev/null -s --max-time 10 \
  -w 'code=%{http_code} dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  https://api.anthropic.com/v1/models
```

A 401 when you send no API key is normal. It means DNS, TCP, TLS and the vendor's gateway are all working. A timeout, 000 or 5xx indicates a real problem. We suggest alerting only after 2 consecutive failures to avoid false alarms from a single network blip.

To measure TTFT, add a second probe: a fixed small request with a key, streaming enabled, recording the time the first data event arrives. Note that curl's `time_starttransfer` records the first response byte, which may be just the headers rather than the first token. To measure TTFT strictly you need to parse SSE events in your script.

Use at least two probe nodes: one near your servers and one near your main users. For a fuller alerting setup see [How to monitor AI app availability](/en/blog/ai-app-availability-monitoring-guide), and for scripts and definitions see [Measuring AI API uptime yourself](/en/blog/measure-ai-api-uptime-yourself).

### Step 2: Add RUM instrumentation to streaming requests in the frontend

Wrap the place where a chat is started with timing and report the result via `sendBeacon`:

```js
async function chatWithTiming(url, body, onChunk) {
  const t0 = performance.now();
  let ttft = null, status = 0, error = null;
  try {
    const res = await fetch(url, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify(body),
    });
    status = res.status;
    const reader = res.body.getReader();
    for (;;) {
      const { done, value } = await reader.read();
      if (done) break;
      if (ttft === null) ttft = performance.now() - t0;
      onChunk(value);
    }
  } catch (e) {
    error = e.name;
  } finally {
    navigator.sendBeacon('/rum', JSON.stringify({
      status,
      error,
      ttft_ms: ttft === null ? null : Math.round(ttft),
      total_ms: Math.round(performance.now() - t0),
      net: navigator.connection ? navigator.connection.effectiveType : null,
    }));
  }
}
```

The server only needs to store these fields and aggregate success count, failure count and TTFT percentiles by hour. Do not report prompts or answers. If you need a region dimension, resolve the IP to a coarse region on the server and then discard the raw IP.

### Step 3: Add a user feedback control

Put two buttons next to each answer, "bad answer" and "slow," and record the time and the model group used. The sample size is far smaller than RUM, but it covers the quality problems that the first two steps cannot see.

### Step 4: Put the three signals on one chart

When an alert fires, read the signals in this order:

1. Synthetic probes failing and the RUM failure rate also rising: the upstream or your own service has a problem. Check the official status page; for how to read them see [How to read the OpenAI and Claude status pages](/en/blog/how-to-read-ai-status-pages).
2. Probes healthy but RUM failure rate rising: most likely the user's network, a regional route, or real concurrency triggering rate limits. Group by error code and region.
3. Both healthy but user feedback increasing: consider a quality problem. Run the fixed-prompt checks and see whether the model or provider changed.
4. Probes failing but RUM healthy: suspect the probe node's own network first, or that the endpoint you probe differs from the one users actually use.

## How does cdnping combine them?

cdnping itself is a combination of three kinds of signals, and can serve as a reference example:

- Edge synthetic probes: [/status](/en/status) probes each official API endpoint from the Cloudflare edge every 5 minutes, syncs selected official status pages every 15 minutes, and keeps 30 days of history;
- Active measurement on the user side: the [home page speed test](/en) measures connection and round-trip time in the user's browser, and the result can be counted anonymously without recording the IP. It sits between synthetic and RUM: the requests are synthetic, but they happen on the user's real network;
- User signals: the one-click reports on [/down](/en/down) fill in problems that probes cannot see, such as "dumber" and "quota reset."

How each data set is collected and defined is documented on the [methodology](/en/methodology) page. Note that cdnping started collecting on 2026-10-17, so the probe and request data cited in this article cover only a very short window. They are used to illustrate the method and do not represent the long-term performance of any service.

## Common mistakes

- Doing only probes and assuming the user experience is fine. A probe only proves that the probe location can connect. It knows nothing about which network the user is on or what device they use.
- Looking only at RUM averages. AI request latency correlates strongly with output length, so long answers pull the average up. Track TTFT and total time separately, and look at p95.
- Treating 401 and 429 as outages. A 401 is usually a key or configuration problem, and a 429 is rate limiting. They are handled differently, so do not merge them with 5xx into a single failure rate. For handling 429, see [What to do when the API returns 429 Too Many Requests](/en/blog/llm-api-429-too-many-requests-fix).
- Using RUM to judge availability at off-peak hours. A percentage computed from a handful of requests swings wildly. In that period, rely on synthetic probes.

A small team does not need to do everything at once. Do steps 1 and 2 above, synthetic probes and streaming request instrumentation, and you will already see network failures, upstream 5xx errors and slow first tokens. Then add the feedback control from step 3 to cover quality problems.