Repository navigation
CI: publish a weekly flaky test dashboard ("Soigneur") - #5201
Merged
Merged
Conversation
A test that fails once in a while, in one of the dozen environments an integration suite runs in, is only visible to whoever happens to look at that one job. Nothing accumulates, so nothing gets fixed. Soigneur is a dashboard for those, with no infrastructure behind it: the archived logs of the analyzed runs are the storage. A soigneur is the keeper who tends the animals, and this one keeps an eye on what Tigron runs: it has two halves, and one marker line per test is the contract between them (see lib.sh): - `flaky-annotate.sh` classifies the tests of a gotestsum json file at the end of a test job, and writes them to the job log as "flaky-test-dashboard: failing TestFoo" (failed, and never passed, even on retry) or "flaky-test-dashboard: flaky TestFoo" (failed, then passed when `--rerun-fails` retried it: flaky beyond a doubt). Both classes are also emitted as annotations, for the humans reading a pull request. - `flaky-report.sh` lists the workflow runs of a branch and, for each of them, downloads the archived logs of all of its jobs in a single request and reads the markers back; `flaky-report.jq` renders the aggregate as markdown; `flaky-issue.sh` publishes it to the one issue whose number it was given, rewriting its description and posting the digest as a comment. It never opens an issue, which is the point: there is exactly one dashboard, and it is the one that was named. Occurrences are deduplicated per (commit, job configuration, kind, top-level test), so that a single bad job execution counts once, and subtests do not inflate the ranking of their parent. Collecting is deliberately per run, and not per job: a run's logs come as one zip, so a week of a busy branch costs about 200 API requests and 50 MB, against the 1000 requests per hour and per repository that the GITHUB_TOKEN of a workflow gets. Reading the annotations of each job through the checks API instead would cost some 800 requests for the same week. The markers are what gets collected, rather than those same annotations, because the runner mangles a multi-line annotation into orphaned fragments in the log. GitHub's job summaries, which do aggregate all of this per run in the web UI, cannot be used at all: they are not exposed by any API, and are rendered from signed attachments only a browser session can mint. The logs of the runs that predate the markers only hold the readable block that flaky-annotate.sh prints at the end of a job, so that block is read as a fallback, and a dashboard has history from the start rather than after a full window. `action.yml` wraps the collecting half as a composite action, so that other projects can use it as-is: everything project-specific is an input, including the caveats rendered in the report's methodology. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Use the mod/soigneur action, on both ends: - gotestsum-reporter.sh hands its json file to flaky-annotate.sh, so that the tests that recovered on retry are reported too. That set was already computed, only to be excluded from the failing ones, and then thrown away - yet those are the tests that are flaky beyond a doubt, and the ones nobody could see, since a job that recovers is green. The step summary now lists them next to the failing ones. - the flaky-test-dashboard workflow collects the week, every Monday on a quiet hour, and publishes it to containerd#5202, the issue it names. It only looks at the runs of the workflows that do run tests, and can also be dispatched manually, with a window, a branch, and a dry-run switch. docs/testing/flaky.md covers what the dashboard reports, what it does not (only the integration suites report per-test data), how to run the collector locally, and how to reproduce, fix, or quarantine a flaky test. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
AkihiroSuda
commented
Sep 10, 2026
| @@ -0,0 +1,188 @@ | |||
| # Soigneur | |||
Member
Author
There was a problem hiding this comment.
cc @apostasie who named Tigron (#3889 (comment)), do you have a suggestion about the name?
Contributor
There was a problem hiding this comment.
This is great to see this!
Soigneur is not bad, especially if you plan in evolving this further into a system that could actively manage flakes.
Alternatively, if you want to stay with animals (aligning with tigron), I would probably go with oxpecker (a small bird that lives on larger animals and pick out their parasites / flakes).
remora (a fish) would be in the same vein.
If you want out of animals, I might go with heisen (from heisenbug).
Contributor
There was a problem hiding this comment.
Good to see you @AkihiroSuda !
I miss being here...
Member
Author
There was a problem hiding this comment.
Hi, good to see you @apostasie
Alternatively, if you want to stay with animals (aligning with tigron),
I think I want something that monitors animals, so I still prefer "Soigneur" 🙂
"heisen" is also nice, but probably conflicts with something else
AkihiroSuda
marked this pull request as ready for review
September 10, 2026 18:04
AkihiroSuda
changed the title
ktock
approved these changes
Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flaky test dashboard (Soigneur)
A test that fails once in a while, in one of the dozen environments the integration suite runs
in, is only visible to whoever looks at that one job. Nothing accumulates, so nothing gets fixed.
This adds a single, long-lived issue - #5202, named by number in the workflow - whose description
is rewritten every Monday with what the CI reported over the last week, plus a one-line digest as
a comment. Nothing ever opens an issue: there is exactly one, and it is the one named.
No database, no artifact, no server: the archived logs of the analyzed runs are the storage.
Real output, over two days of
main, trimmed - the report also lists every job configurationthat failed, and links every occurrence to its job:
TestLoadStdinFromPipeTestAttachForAutoRemovedContainergotestsum-reporter.shhands its json file toflaky-annotate.sh, which writes one marker lineper test to the job log, including the tests that passed when
--rerun-failsretried them - aclass that was already computed, then thrown away, yet the only one that proves flakiness.
Weekly, the collector lists the workflow runs of the branch and downloads the logs of each run in
a single request to read the markers back. Per run rather than per job: ~210 API requests a week
instead of ~880 through the annotations of the checks API, against a budget of 1000 per hour.
Everything reusable lives in
mod/soigneur- Soigneur, a composite action; a soigneur isthe keeper who tends the animals, and this one keeps an eye on what Tigron runs. See
docs/testing/flaky.mdandmod/soigneur/README.md.Notes for review
Recovered on retryreads 0 until this merges: the markers are not in the archived logsyet. Everything else works on existing logs, because the collector also reads the
=== Failing tests ===block the reporter has always printed - that produced the table above.contents: read,actions: read,issues: write; never runs on a fork. Test names comefrom job logs, so implausible ones are dropped and the tables are escaped.
executions, 18 MB, under 100 s) and cross-checking one job against the checks API.
make lint-shelland a localzizmorv1.30.1 are clean;make lint-yamlwas not run locally.Actions>flaky-test-dashboard>Run workflow,with
publishunchecked.🤖 Generated with Claude Code
Preview
Flaky test dashboard
Failures reported by the CI on
mainbetween 2026-09-08 and 2026-09-10 — 10 commits with CI results, 354 job executions.Tests
TestLoadStdinFromPipeTestAttachForAutoRemovedContainerTestRunRmTimeTestBuildContextWithOCILayoutTestComposePsJSONTestComposeUpWithExternalNetworkTestContainerListWithFilter3 casesTestIPFSAddrWithKuboTestLoadQuietTestRunContainerWithStaticIP63 casesTestRunDomainname2 casesTestSave2 casesTestSaveContentTestSaveQuietTestSyslogNetwork4 casesTestLoadStdinFromPipe— 3 occurrencesin-host / rootless-port-slirp4netns linuxTestLoadStdinFromPipein-host / nerdctl.gomodjail < rootless linuxTestLoadStdinFromPipein-host / rootless-port-slirp4netns linuxTestLoadStdinFromPipeTestAttachForAutoRemovedContainer— 2 occurrencesin-host / docker < linuxTestAttachForAutoRemovedContainerin-host / docker < linuxTestAttachForAutoRemovedContainerTestRunRmTime— 2 occurrencesin-host / rootless-port-slirp4netns linuxTestRunRmTimein-host / rootless linuxTestRunRmTimeTestBuildContextWithOCILayout— 1 occurrencein-host / rootful linuxTestBuildContextWithOCILayoutTestComposePsJSON— 1 occurrenceEL / almalinux-8 rootfulTestComposePsJSONTestComposeUpWithExternalNetwork— 1 occurrencein-host / rootful linux (old ubuntu) (ctd: v1.7.34)TestComposeUpWithExternalNetworkTestContainerListWithFilter— 1 occurrenceEL / almalinux-8 rootlessTestContainerListWithFilterTestIPFSAddrWithKubo— 1 occurrencein-host / rootful linuxTestIPFSAddrWithKuboTestLoadQuiet— 1 occurrencein-host / nerdctl.gomodjail < rootless linuxTestLoadQuietTestRunContainerWithStaticIP6— 1 occurrenceEL / almalinux-8 rootfulTestRunContainerWithStaticIP6TestRunDomainname— 1 occurrencein-host / rootful linux (old ubuntu) (ctd: v1.7.34)TestRunDomainnameTestSave— 1 occurrencein-host / rootful linuxTestSaveTestSaveContent— 1 occurrencein-host / rootless linux (arm)TestSaveContentTestSaveQuiet— 1 occurrencein-host / rootless linuxTestSaveQuietTestSyslogNetwork— 1 occurrenceEL / almalinux-8 rootfulTestSyslogNetworkJob configurations
Every job, test or not: a job that fails without any test failing is usually an environment, timeout, or infrastructure problem.
in-host / windows████░░░░░░40%in-host / rootful linux███░░░░░░░30%in-host / rootful linux (old ubuntu) (ctd: v1.7.34)███░░░░░░░25%EL / almalinux-8 rootful██░░░░░░░░20%in-host / docker < linux██░░░░░░░░20%in-host / nerdctl.gomodjail < rootless linux██░░░░░░░░20%in-host / rootless linux██░░░░░░░░20%in-host / rootless-port-slirp4netns linux██░░░░░░░░20%EL / almalinux-8 rootless█░░░░░░░░░10%FreeBSD / FreeBSD█░░░░░░░░░10%in-host / rootless linux (arm)█░░░░░░░░░10%How this is collected
This dashboard is generated by
Soigneur,
which aggregates the GitHub Actions annotations that the test jobs attach to themselves.
No database, test artifact, or server is involved, so the window that can be reported on is
limited by the retention of the check runs (90 days, by default).
(
gotestsum --rerun-fails). Those are flaky beyond a doubt. The others are either flaky,or consistently broken.
FreeBSD do not run the reporter.
See https://github.com/containerd/nerdctl/blob/main/docs/testing/flaky.md to reproduce, fix, or quarantine a flaky test.
Generated by this run.