Tracking Issue for libtest's JUnit reporter

[andoriyu]

This is a tracking issue for libtest's JUnit reporter.

Add an alternative formatter to libtest. The formatter produces valid xml that later can be interpreted as JUnit report. Report can be consumed by Continuous Integration tools like Jenkins.

Public API

Run test binary with --format=junit argument.

Steps / History

• Implementation: feat(libtest): Add JUnit formatter #84568 Use as_secs_f64 in JunitFormatter #86359
• Final commenting period (FCP)
• Stabilization PR

Unresolved Questions

• timestamp is required by schema, but every viewer/parser ignores it. Attribute is not set to avoid depending on chrono;
• Each test suite (doc tests, unit tests and each integration test) must be run separately. This due to a fact that from libtest perspective each one of them is a separate invocation.
• libtest doesn't know what is the name of integration binary is being run, so in the report it just says integration for all of them.
• No test cases in src/test or otherwise validating output format correctness.

[jonasbb]

rust/library/test/src/formatters/junit.rs

Lines 77 to 83 in 9f83e22

	self.write_message(&*format!(
	"<testcase classname=\"{}\" \
	name=\"{}\" time=\"{}\">",
	class_name,
	test_name,
	duration.as_secs()
	))?;

It would be nice if the output supports sub-second precision for the duration. This is what cargo2junit does. That way you don't lose the runtime information for fast tests.
I just repeat the comment here if my message on #84568 gets lost.

[spencergilbert]

Also commented on #49359 - but we'd love to help stabilize either of these, primarily to support Datadog CI

[andoriyu]

rust/library/test/src/formatters/junit.rs

Lines 77 to 83 in 9f83e22

	self.write_message(&*format!(
	"<testcase classname=\"{}\" \
	name=\"{}\" time=\"{}\">",
	class_name,
	test_name,
	duration.as_secs()
	))?;

It would be nice if the output supports sub-second precision for the duration. This is what cargo2junit does. That way you don't lose the runtime information for fast tests.
I just repeat the comment here if my message on #84568 gets lost.

Done via #86359

[cemoktra]

Hey. What is timeline to stabilize this?

[last-partizan]

Each test suite (doc tests, unit tests and each integration test) must be run separately. This due to a fact that from libtest perspective each one of them is a separate invocation.

Any ideas how to approach this?

[andoriyu]

Each test suite (doc tests, unit tests and each integration test) must be run separately. This due to a fact that from libtest perspective each one of them is a separate invocation.

Any ideas how to approach this?

Currently — just do multiple cargo invocations for each.

[last-partizan]

I mean, in the long run, from architectural point of view. Looks like there is no way to combine all xml output into single file at libtest level, becouse each invocation is separate. So, maybe it's better to move formatters into cargo-test? Or maybe there is a way to signal global test start to libtest, so it can write header, and then after all test finished - write footer?

[detly]

Since both this and the JSON output is now gated as of 1.70, I just wanted to add a couple of notes from a user:

For anyone looking into how to get jUnit reporting for Rust projects now and finding this, we migrated our Gitlab CI to cargo-nextest instead of pinning to 1.69 or using RUSTC_BOOTSTRAP etc. We did not have any major issues with it, the docs are good and it took a couple of hours. YMMV of course. (Sorry if anyone sees this twice, I cross posted on another linked issue but it might help others finding this for the first time.)

Regarding this:

Each test suite (doc tests, unit tests and each integration test) must be run separately. This due to a fact that from libtest perspective each one of them is a separate invocation.

Any ideas how to approach this?

Currently — just do multiple cargo invocations for each.

I think this is a big reason that this feature has had less attention than the JSON output, and people seem more concerned with the JSON-output-via-cargo2unit workflow breaking than this needing completion/stabilisation. I myself am guilty of looking into CI integration a couple of years ago when we started with Rust, finding this very issue, thinking "this won't work" and then going off to suity/cargo2junit without leaving any feedback. So here it is.

It seems like a little thing - just run cargo test multiple times, upload multiple reports, etc. But there are a few problems with that, that add up to quite a maintenance burden:

• Gitlab CI at least (I don't know about others) will treat and render separate reports differently to one report containing the invocations as testsuites. It is more nested, more annoying to navigate, more difficult to compare if names or (module) paths change etc. (I admit I have not checked in on this in a long time.)
• It also means you have to write your CI config to "know" what cargo test invocations to do. We rely a lot on integration tests. It's the sort of thing I'd want a test harness (or the thing that calls it) to take care of automatically.
• You have to write CI config/scripts to either know where are all the reports are (duplicating the information in the invocation step), or some script that combines them all into one (please, no).
• Most annoyingly, there is no way to make this generic! If you have multiple projects, you can't just have a CI template/workflow that does cargo test in the project dir.

If it's an architectural issue, then maybe it is necessary to rely on JSON output for jUnit reporting, or see if it can go in Cargo instead.

[therealfrauholle]

Thank your for bringing this up @detly. I share your view and pain points. It makes me also very sad that I noticed a bug in the JUnit formatter (see #99813) but never cared enough to fix it. We now use gitlab-report, an alternative to cargo2junit to produce junit test reports in our pipeline.

I did not bother to fix #99813 because it feels really bad to implement XML generation by hand if technically good crates already exist for that. Generation might break for something else in the future and then we need to waste time again. gitlab-report gives me the freedom to have fancy macros and complex handling, being able to test functionality without going through the rust review process or complicating my test environment (e.g. building the rust toolchain myself).

This makes me wonder: Do we actually want to maintain a junit reporter after all? I have the feeling it would be better if the libtest component focuses on providing a rich API to implement custom reporters. I see that the json output format starts to be that default API, so from the view of the rust core language it might be smart to drop support for junit and other output formats in libtest and focus on how we can ease the work for tools like cargo nextest, gitlab-report or cargo2junit; e.g. stabilize the/a json output format.