`x test` on profiles where testing is unsupported (or unmaintained?) could set better expectations

[steffahn]

Context

If you run test suites with x.py test (sub)command(s) in environments we don't run it in in CI, they may fail. That's natural and expected, of course, but the tooling output doesn't make it particularly clear what usage is or isn't intended.

If the user is on a weird platform or such, this might be expected anyway; it becomes surprising if the thing that caused a faulure is merely a different bootstrap.toml setup. Most notably if the setup happens to be exactly one of the presets that x.py setup offers.

As an extreme example of the confusion this may lead to: A plain x.py test without any extra arguments, with a basic profile = "dist" profile (created in x.py setup), on a normal x86_64-unknown-linux-gnu (Ubuntu) setup, used to¹ be prominently displaying an ICE error message as part the first failing test.

• UI tests fail on dist profile in confusing manner (making it look almost like an ICE happened) #158219

An initial search for similar/related kinds of issues had me finding

• ./x test library/std fails with library profile #142505

which seems to also fit this category of problems (though in this case, it's more obviously a case where things should actually just be fixed).

Questions

On one hand, I have some remaining questions about the dist profile: Is the dist profile in particular even meant to be used with commands like x test? I also wonder (i.e. haven't quite figured out yet) what about it makes it more prone to test failures in particular?

---

On the other hand, I would be interested in judging potential ways forward so that someone who sees a test failure wouldn't even need to wonder about such fundamental things as “is this supposed to work in the first place?”.

👉 expand/collapse the list - [I’m trying not to take away all the focus from the first question]

1. certain profiles could "disable"² certain x.py subcommands that aren't intended/meant to work anyway, to stop them from running before they even begin

   • this would make the failure quicker, and the error message more consistent than "whatever happens to be the first thing that breaks"
   • the error would provide information on how to change bootstrap.toml, or maybe pass an extra argument, to run the test anyway

2. or instead, nothing should change - reasons for this may include:

   • there's not really any issue; the test suite is mostly for our internal usage anyway
   • it's too much work to implement and maintain this (for too little benefit)
   • we need better documentation for contributors, and with better documentation, things will be clear enough
   • x test shouldn't fail on dist in the first place because [xyz], we should just fix this

3. or instead, certain profiles could emit extra warnings/notes with certain x.py subcommands

   • warnings can work similar to out-of-date-change-id warnings, showing both at the start and end of invocation
   • could set expectations through case-specific messages (document known/expected failure; point to context/workarounds for known issues³; or conversely possibly encourage reporting issues for cases that really should work [but not enforced by CI])

4. or instead, something else? Or a combination of things?

Footnotes

1. at the time of this writing waiting for the merge queue ↩

2. that is, always produce an error message when it's used ↩

3. it would ideally be easy to configure certain configuration+command usage patterns, if "known issues" (like ./x test library/std on library profile) should be handled, so contributors can keep them up-to-date easily ↩

[bjorn3]

The dist profile is meant to be used by distros, who do often run ./x.py test as part of building their rust package.

[Kobzol]

bootstrap.toml allows arbitrarily changing how the build process and bootstrap works, including setting compiler flags, changing the C/C++ compiler, etc. You can break your build in an infinite number of ways just by changing bootstrap.toml. And to exactly reproduce what we do on CI, you'd need to use the same OS, same bootstrap.toml configuration, etc.

So I wouldn't say that specific default profiles are unsuitable for specific x commands, it's more that there are simply bootstrap.toml options that will break some of those commands.