Blog / Troubleshooting

Incident Postmortem Template for API Outages (With Timeline Data)

2026-10-25cdnping editors7 min read

A usable incident postmortem template has six core parts: summary, impact, timeline, root cause, contributing factors, and action items (the template below adds a seventh section, "What went well"). Two of them tend to come out vague. The timeline should be verified against monitoring data, not stitched together from memory. And when the failure is upstream (say, an AI model API), the subject of the review shifts to "how we responded." Below is a template you can copy, followed by notes on making each section concrete.

#Why does nobody read most postmortems?

Three common problems:

• The timeline was reconstructed from memory afterward ("we noticed something around 3 o'clock") and nobody can check it.
• The root cause is "the upstream service was unstable," with nothing after it, and the action item is "improve monitoring."
• Impact is "some users were affected," with no request counts, failure rate, or duration.

The point of a postmortem is to spend less time on the next incident. Every sentence should answer: where is the evidence, and could we detect this earlier or recover faster next time?

#Incident postmortem template (copy and paste)

# Postmortem: <one-line symptom>, <date>

Status: Draft / In review / Final
Owner: <role, e.g. on-call that week>
Review date: <date>

## 1. Summary
- What happened (one sentence):
- How long (start - recovery, with time zone):
- How big (failed requests / failure rate / affected features):
- Root cause category: own code / config / infrastructure / upstream dependency / network
- Current status: recovered / mitigated, permanent fix pending

## 2. Impact
| Metric | Value | Data source | Window |
|---|---|---|---|
| Failed requests | | | |
| Failure rate (failed / total requests) | | | |
| Affected features or endpoints | | | |
| User-visible duration | | | |

## 3. Timeline (use UTC throughout; add Beijing time in parentheses if your team works in China)
| Time | Event | Evidence (link / screenshot / log query) |
|---|---|---|
| T0 | First anomaly signal (probe failure, error rate rise) | |
| T1 | Alert fired | |
| T2 | Someone acknowledged and started working on it | |
| T3 | Upstream status page first updated (if applicable) | |
| T4 | Mitigation applied (failover, degradation, rollback) | |
| T5 | Metrics recovered | |
| T6 | Vendor declared recovery / incident closed (if applicable) | |

Key intervals: time to detect T1-T0 = ; time to respond T2-T1 = ; time to mitigate T5-T2 = 

## 4. Root cause
<What was the direct cause? Back it with evidence. If unsure, write "to be confirmed" and say how it will be confirmed.>

## 5. Contributing factors
<What made the impact bigger or longer: missing timeouts, retry amplification, single vendor, alert thresholds, etc.>

## 6. What went well / where we got lucky
-

## 7. Action items
| ID | Action | Type (prevent / detect / mitigate) | Owner | Due date | Acceptance criteria |
|---|---|---|---|---|---|
| A1 | | | | | |

A few conventions:

• Blameless: write "the alert threshold was set to 30 minutes, which delayed detection" (an example), not "so-and-so didn't look at the alert." Review the system, not the person; otherwise people will write fewer details next time.
• One time zone: status pages, logs, and monitoring platforms often use different time zones, and mixing them can shift a timeline by hours.
• Every action item needs acceptance criteria. "Improve monitoring" is not an action item. "Add an external probe for the upstream API every 5 minutes and alert after 2 consecutive failures" (tune the numbers to your business) is.

#How do you rebuild the timeline from data instead of memory?

The most valuable parts of a timeline are the gaps: when the problem started, when you found out, when the vendor acknowledged it, and when it was really over. Each comes from a different data source, and they need to be lined up one by one.

#Step 1: Pin down T0, the first anomaly signal

Take T0 from the first failure in an external probe or request log, not from the alert time. Alerts have aggregation windows and thresholds, so they fire later than the real start.

If you have your own request logs, count 5xx responses and timeouts per minute:

# Assumes JSON-lines logs with ts (ISO timestamp) and status fields
jq -r 'select(.status >= 500) | .ts[0:16]' app.log | sort | uniq -c | head -20

The first minute clearly above baseline is your candidate T0. Look back another 10 minutes to confirm things were normal before it.

If you have no probes of your own, a third-party external probe can help. The cdnping API status page probes each vendor's official API endpoint from the Cloudflare edge every 5 minutes and keeps 30 days of history; the probing method is described in Methodology. Note that cdnping only began collecting on 2026-10-17, so earlier incidents have no probe data from it. A 5-minute resolution is enough to check whether the endpoint was reachable during a window, but minute-level precision has to come from your own logs. To build your own probes, see Measure AI API uptime yourself.

#Step 2: Match against the vendor's status page

The "incident created" time on an upstream status page is T3, and it is often later than T0. For how to read incident levels and components, see How to read the OpenAI and Claude status pages.

Below is a worked example using official status-page records synced by cdnping (source: official status-page incidents that cdnping syncs every 15 minutes, a few entries around 2026-10-15; the minutes are the incident durations as recorded by cdnping):

Suppose your service depends on the OpenAI API and was affected by the October 15 incident. The timeline could read:

T0  2026-10-15 16:2x UTC  5xx starts rising in our own logs (fill in the actual minute)
T3  2026-10-15 16:29 UTC  OpenAI status page creates incident "Widespread feature degradation" (major)
T6  about 18:55 UTC       incident close time, derived from the recorded 146-minute duration
T5  ?                     our error rate returns to baseline (per our logs; may be before or after T6)

Keep T5 and T6 separate. The time the vendor closes an incident and the time your service recovers are not necessarily the same: your retry queue may still be working through a backlog, or you may have switched to a fallback path long before.

#Step 3: Check whether the data itself is trustworthy

The last row of the table shows a duration of -43 minutes. Other entries in this cdnping record set also have negative values (for example -57, -17, and -183 minutes). Possible causes are that the status page rewrote the start time after the fact, or that the incident was backfilled. Whatever the reason, never put a negative duration straight into a postmortem. When you see one, go back to the incident detail on the official status page, read the update times for each "Investigating / Identified / Resolved" entry, and compute it yourself.

This applies to your own monitoring too: a dead probe node, clock drift, or delayed log ingestion can all skew a timeline. State the source of every timestamp so reviewers can check it.

#Step 4: Quantify impact with request counts

The impact section works best with a denominator. As an example, take the hourly request statistics published by AGI Plan (AGI Plan is a commercial partner of cdnping; the data comes from https://agiplan.dev/api/status/requests, requires no login, and counts only 5xx as platform failures). Here are three consecutive hours from 2026-10-16, with each hourly bucket starting at minute 5 past the hour:

Data like this answers "how big was the impact and when did it start to ease," but not "why." Whether the upstream had an incident in the same window has to be checked against status-page records; these three rows alone do not support a root-cause conclusion. That is exactly why "root cause" and "impact" are separate sections.

For converting failure rate, availability, and error budget, see SLI, SLO, and SLA: what's the difference.

#When an upstream AI service fails, what goes in your own postmortem?

When the root cause is "OpenAI / Claude / some cloud service had a problem," many teams skip the postmortem. They shouldn't, because upstream failures are not rare. According to the official status-page records synced by cdnping (2026-09-17 to 2026-10-17), Claude had 6 major incidents, OpenAI had 4 major and 19 minor, and Cloudflare had 3 major and 19 minor. For fuller statistics see AI service incidents: September-October 2026.

You can't change the upstream root cause, so focus the postmortem on these questions:

#Did we learn about it before or after the vendor?

Compare T0, T1, and T3. If your alert fired after the vendor's status page updated, you are relying on user reports or vendor announcements to detect problems. The fix is external probing; see How to monitor AI application availability. When something looks wrong, you can also check /down first. It aggregates official status pages, API probes every 5 minutes, and user reports, so you can quickly tell whether the problem is widespread.

#Did we handle error codes correctly?

Look at the status code distribution of failed requests in your logs:

jq -r 'select(.upstream=="openai") | .status' app.log | sort | uniq -c | sort -rn

Codes like 502, 503, and 529 are usually worth retrying with backoff; 400 and 401 are not, and retrying them only amplifies traffic. If retries multiplied your request volume several times during the incident, that is a contributing factor. For what each status code means and whether to retry, see AI API error codes 400 to 529 cheat sheet.

#Did we have a degradation or failover path, and did we use it?

Check each item:

• Is there a backup model or backup vendor? Does switching require a code change or a config change?
• How long did the switch take? Who decided? Were the criteria written down in advance?
• What did users see after degradation: errors, a queue notice, or features silently getting worse?
• Are timeouts reasonable? Did hung requests fill the connection pool and take down other features?
• Did we switch back after recovery? Did switching back cause a second blip?

#What did we tell users?

Record the time, channel, and wording of the first external announcement. If users noticed the problem before your announcement, consider building a status page for your own service; see Build a status page for your AI service.

#Example action items for an upstream-failure postmortem

The "configured multiple" and "10 minutes" in the table should be set for your own business; they are examples only.

#Self-check checklist before you submit

Run through this before the review:

1. Does every timeline row have an evidence link or a query?
2. Is the time zone consistent?
3. Does the impact section have a denominator (total requests) and a window?
4. Is the root cause backed by evidence or is it a guess? Are guesses marked "to be confirmed"?
5. Are any individuals blamed by name? Rewrite as a description of process and systems.
6. Does every action item have an owner, a due date, and acceptance criteria?
7. Is there a separate answer to "if this happened again, would we detect it sooner?"
8. Do cited external data (status pages, third-party probes) state their source and window, and were they checked for outliers (such as negative durations)?

After the postmortem is finalized, check back in a week or two to see how many action items are done, and go through the unfinished ones separately in your weekly meeting.