Failure rate spike alerts for CI test runs

TestNod's failure rate spike detection watches what share of your tests fail on every CI run and fires an alert when a run comes in well above the recent average. Where flaky test detection tracks individual tests that pass and fail without any code changing, this alert looks at the suite as a whole, so it catches the run where a regression landed and a batch of tests went red together. The check runs automatically against every new test run that finishes with at least one failure or error.

How TestNod detects a failure rate spike

When a test run finishes processing, TestNod takes the 20 most recent processed runs in the project that share the same set of tags as the new run, not counting the new run itself. For each of those, it computes a failure rate as a percentage:

failure_rate = ((failures + errors) / total_tests) * 100

Errors count alongside failures on both sides of the comparison, since a test that blew up before it could assert anything is as broken as one that failed an assertion. Runs that reported no tests at all are excluded, because there is no rate to draw from them. TestNod averages the remaining rates into a baseline, computes the same rate for the new run, and the alert fires when:

current_failure_rate - baseline_failure_rate >= threshold_percentage

The default threshold is 15 percentage points. For example, a suite that normally fails at 2% trips the alert on a run that comes in at 17% or worse.

Why the threshold is measured in percentage points

The threshold is a difference, not a multiple. Doubling a 1% failure rate is noise, while doubling a 30% rate is a crisis, so a single multiplier cannot serve both. Measuring the gap in percentage points keeps the setting meaningful wherever your suite normally sits, and it still behaves sensibly for a suite that is usually all green, where any relative increase over a 0% baseline would be infinite.

The 20-run baseline for failure rate analysis

Failure rate spike detection needs at least 20 processed runs on the same set of tags before it can fire on a project. Below that, TestNod skips the check and no event is recorded, because an average drawn from a handful of runs moves too much to be worth comparing against. Once enough history accumulates on a given tag combination, every subsequent run is evaluated against it.

The triggering run is left out of its own baseline, so the run being judged never raises the bar it has to clear.

Runs that reported no tests at all are dropped from the average. A few of those in the window are fine and the alert still fires, but if fewer than 15 of the 20 came back with results, TestNod skips the check rather than average over a thin sample.

Example: detecting a spike in CI test failures

The following example uses a project that tags every CI run with ci and integration and leaves the threshold at the default 15 percentage points. Across the last 20 runs on that tag set, roughly 2 tests out of 100 failed each time, giving a baseline failure rate of 2%.

The next run reports 15 failures and 5 errors out of 100 tests, a failure rate of 20%. The gap is 18 percentage points, which clears the threshold, so the alert fires and records the run's rate alongside the 2% average it was measured against.

What the alert event shows

Each occurrence on the alert detail page reads as a single line covering the rate the run came in at, the average it was measured against, how many recent runs went into that average, and how many of the run's tests failed. When the recent runs were all green, the line reports that the rate jumped after a stretch with no failures rather than quoting a 0% average.

Every occurrence links back to the test run that triggered it, where the What Changed panel lists which tests started failing. The tag set behind the comparison is shown near the top of the alert page.

Tuning the failure rate threshold

The right threshold depends on how noisy your suite already is:

• 5 points is aggressive and suits a suite that is reliably green, where any cluster of new failures is worth a look. Expect more alerts if your suite has flaky tests.
• 15 points (default) is a reasonable starting point for most projects. It ignores day-to-day wobble and fires when a meaningful share of the suite breaks at once.
• 25 points or higher only catches dramatic breakage. Reach for it when a suite is noisy enough that smaller jumps aren't actionable, though fixing the noise is usually the better trade.

If your recent runs have been entirely green, the baseline sits at zero and the threshold becomes the plain failure rate that trips the alert. For example, a project left on the default fires the first time 15% of the suite goes red. That is usually what you want from a suite with a clean history, but it's worth knowing before you set the number low.

Project admins set the threshold from the alert configuration page, where the field is labeled Threshold percentage. For this check, read the number you enter as the size of the jump rather than as a failure rate on its own. See Configuring alerts for the walkthrough.

What the alert does not catch

• Projects without 20 same-tag runs yet. The check needs a full window on the matching tag set before it can fire, so new projects and new tag combinations stay quiet until the history fills in.
• A failure rate that climbs slowly. This alert fires on the jump, not the level. If a suite degrades a little each week, each run sits close to its own baseline and nothing fires, even though the suite has drifted badly over a month. Skipped test creep covers the equivalent slow drift for skipped tests.
• A failure rate that stays high. Once a bad rate persists long enough to become the baseline, the gap closes and the alert stops firing. That is deliberate, since a sustained rate is no longer a spike, but it does mean the alert tells you when the build broke rather than nagging you until it's fixed.
• A single test that fails intermittently. That is what flaky test detection is for.

Where to start

Once your project has 20 same-tag runs of history, an admin can enable the failure rate spike alert on the alert configuration page and set the threshold. Every test run after that is compared against its rolling baseline, and when a run clears the threshold an event is recorded on the alert detail page and a notification email goes to project members who have not opted out.

If you haven't sent your first test run yet, the Quickstart walks through generating JUnit XML reports and uploading them from your CI in about five minutes.