Track flaky test retries and reruns in CI

When a test fails and then passes on a later attempt in the same CI build, TestNod marks it passed on retry and keeps the failure on the test. The build result stays passing, because that's what your CI reported, but the flakiness alert and the flakiest tests report count the retry as a flaky result. The What Changed panel lists the test under "Needed a retry".

A plugin like rspec-retry reruns a failing test inside the same process, so the JUnit report only has the result of the last attempt. If you rerun failed tests as a separate step and upload that report as its own file, TestNod gets both attempts.

How TestNod finds retried tests in JUnit XML

TestNod identifies a test by its class name and test name, as described in JUnit XML format notes. If the same test appears more than once in a build and its last attempt passed after an earlier one failed or errored, TestNod keeps a single entry marked passed on retry. That entry holds the failure message from the failed attempt and the number of retries. The earlier attempts come out of the run's totals, so a build whose only failures were retried shows as passed.

The repeated attempts can come from one report file or from two:

• The same report file. When a runner writes a separate <testcase> entry for each attempt in one JUnit XML file, there's nothing to set up.
• A separate rerun file. When you rerun failed tests into a second report, upload that file to the same build_id with "retry": true. Only a file with this flag can replace failures that came from another file in the build.

The flag exists because of matrix builds. When two legs of a matrix share a build_id and a test fails on one leg but passes on the other, neither file is a rerun of the other. TestNod keeps both results, and the build stays failed.

Maven Surefire records its reruns inside a single <testcase> instead of repeating it, and TestNod reads those too. See Maven Surefire flaky test reruns.

Rerun failed tests into a second JUnit report

Each recipe below runs the full suite, then runs a second command that writes only the failures to another file. Let the first test command fail without stopping the job, so the rerun's exit code decides whether the build passes. Then send both reports to TestNod.

Step 1: Upload the first report through your CI's uploader or the submission API, without finalizing the build.

Step 2: Upload the rerun report to the same build with the retry flag set. Each TestNod uploader has a retry: true option for the step that uploads the rerun report, covered on the GitHub Actions, CircleCI, and GitLab CI/CD pages. On GitHub Actions it looks like this:

      - name: Upload rerun results to TestNod
        if: ${{ !cancelled() }}
        uses: testnod/testnod-uploader-action@v1
        with:
          file: tmp/rspec-retry.xml
          token: ${{ secrets.TESTNOD_PROJECT_TOKEN }}
          retry: true
          finalize: false

On a CI provider without an uploader, add "retry": true to the request body of the submission API call:

RESP=$(curl -sS -X POST https://testnod.com/integrations/test_runs/upload \
  -H "Project-Token: $TESTNOD_TOKEN" -H "Content-Type: application/json" \
  -d "{\"retry\":true,\"test_run\":{\"metadata\":{\"build_id\":\"$CI_BUILD_ID\",\"branch\":\"$BRANCH\",\"commit_sha\":\"$SHA\"}}}")
curl -sS -X PUT -H "Content-Type: application/xml" \
  --upload-file retry-results.xml "$(echo $RESP | jq -r .presigned_url)"

Step 3: Finalize the build once with the same build_id, after both uploads. With an uploader, that means finalize: false on both upload steps and a last step with finalize: only.

RSpec --only-failures

RSpec's --only-failures option reruns the examples that failed in the previous run. It needs config.example_status_persistence_file_path set in your spec_helper.rb. The spec_helper.rb that rspec --init generates includes that line inside a commented-out block, so uncomment it if your project hasn't already.

bundle exec rspec --format RspecJunitFormatter --out tmp/rspec.xml || true
bundle exec rspec --only-failures --format RspecJunitFormatter --out tmp/rspec-retry.xml

Upload tmp/rspec.xml as usual and tmp/rspec-retry.xml as a retry, and drop rspec-retry from the suite. If the first command passes, --only-failures has nothing to run and writes an empty report. Uploading an empty retry file changes nothing.

pytest --last-failed

pytest remembers the last run's failures in its cache, and --last-failed (or --lf) reruns only those.

pytest --junitxml=reports/pytest.xml || true
pytest --last-failed --last-failed-no-failures none --junitxml=reports/pytest-retry.xml

--last-failed-no-failures none keeps the second command from rerunning the whole suite when nothing failed the first time.

Playwright --last-failed

Playwright's --last-failed flag reruns the tests that failed in the previous run.

npx playwright test --reporter=junit > results.xml || true
npx playwright test --last-failed --reporter=junit > results-retry.xml

Maven Surefire flaky test reruns

Surefire's rerunFailingTestsCount setting reruns a failed test during the same mvn test run, up to the number of times you set:

mvn test -Dsurefire.rerunFailingTestsCount=2

When a rerun passes, Surefire writes the test once and adds a <flakyFailure> or <flakyError> element for each attempt that failed before it. TestNod marks that test passed on retry with one retry per element, and shows the message and stack trace from the last failed attempt. The report you already upload carries all of this, so you don't need a second file or the retry flag.

If you also rerun failed tests into a separate report and upload it with "retry": true, a test that Surefire marks flaky in that rerun still replaces its earlier failure. Its retry count includes the earlier failure and every flaky attempt in the rerun.

A test that fails on every rerun keeps its <failure> or <error> and gains <rerunFailure> or <rerunError> elements. TestNod records it as failed or errored and doesn't read the rerun elements.

Where retried tests show up in TestNod

• Test suite page: an amber Passed on retry badge, plus a status filter option that lists only those tests.
• Test case page: the failure message from the failed attempt and the retry count, with retried runs counted as passes in the pass rate.

The test run page shows how many tests passed on retry under its test count. The test runs API reports these tests with the passed_on_retry status, and both the API and the test_run.completed webhook include a retried count in the run's totals.