Non-deterministic overflow in rustc_thread_pool causing CI failures

It seems that enabling overflow checks (#89776) uncovered non-deterministic overflows in rustc-rayon-core causing the build failures of #90042 and #90222:

Overflow location:
https://github.com/rust-lang/rustc-rayon/blob/c8ec88d8a2236d1fe19d65a4ab38834f76d256b7/rayon-core/src/sleep/mod.rs#L330

This code is not in upstream rayon but just in our fork, introduced here:
rust-lang/rustc-rayon@27911f7#diff-cde9d726ca2f32c319420be6c61d2e57ad9daff7f85a5f8f7be25474a15f9c4dR328

@rustbot

@rustbot label +T-compiler +A-parallel-queries

Rollup merge of rust-lang#90234 - rusticstuff:rustc-rayon-core-no-ove…

Interestingly enough the errors only appeared on Windows, in rustdoc and in builds which don't have the parallel queries enabled. So I guess Rustdocs use of rustc-rayon triggers it?

rust/src/librustdoc/docfs.rs

Lines 66 to 77 in d1d8145

    
           if !self.sync_only && cfg!(windows) { 
        
               // A possible future enhancement after more detailed profiling would 
        
               // be to create the file sync so errors are reported eagerly. 
        
               let sender = self.errors.clone().expect("can't write after closing"); 
        
               rayon::spawn(move || { 
        
                   fs::write(&path, contents).unwrap_or_else(|e| { 
        
                       sender.send(format!("\"{}\": {}", path.display(), e)).unwrap_or_else(|_| { 
        
                           panic!("failed to send error on \"{}\"", path.display()) 
        
                       }) 
        
                   }); 
        
               }); 
        
           } else {

@jyn514

cc @jyn514 since IIRC you were looking into the DocFS code at some point recently

I don't have any insight here, I think this just happens to be the only part of the compiler using rayon when cfg(parallel_compiler) is disabled.

Hm... I would suggest that rustdoc probably wants to use mainline rayon here, rather than rustc's fork of it, since I/O code likely doesn't want to interop with jobserver and such. I don't know whether that will fix this particular issue, but it seems like a good idea.

@Mark-Simulacrum

@Mark-Simulacrum what jobserver are you referring to here? I don't know what the differences between mainline rayon and rustc's fork are.

The fork is primarily intended to interface with the jobserver that Cargo creates (though I don't know that rustdoc ever configures it as such) to ensure that rayon's worker threads are not using more than the allocated share of CPU resources. This is the same jobserver that manages e.g. LLVM parallelism within each rustc compilation, for example, to ensure there's at most "ncpu" threads running at the same time.

For I/O heavy work like this, I'd expect that you don't want that - your threads are mostly blocked on I/O completion which is not CPU-heavy for the most part.

rustdoc: Switch to mainline rayon

Rollup merge of rust-lang#161296 - aerooneqq:try-enable-overflow-chec…

Unrolled build for #161296

Reopening, this only can be closed if #161296 is not reverted in a month or so.

Rollup merge of #161296 - aerooneqq:try-enable-overflow-checks, r=pet…

Reopening, this only can be closed if #161296 is not reverted in a month or so.

Almost two months have passed.

Navigation Menu

Repository navigation

Non-deterministic overflow in `rustc_thread_pool` causing CI failures #90227

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if !self.sync_only && cfg!(windows) {
	// A possible future enhancement after more detailed profiling would
	// be to create the file sync so errors are reported eagerly.
	let sender = self.errors.clone().expect("can't write after closing");
	rayon::spawn(move \|\| {
	fs::write(&path, contents).unwrap_or_else(\|e\| {
	sender.send(format!("\"{}\": {}", path.display(), e)).unwrap_or_else(\|_\| {
	panic!("failed to send error on \"{}\"", path.display())
	})
	});
	});
	} else {

Non-deterministic overflow in rustc_thread_pool causing CI failures #90227

Description

Activity

hkratz commented on Oct 24, 2021

hkratz commented on Oct 24, 2021

camelid commented on Oct 27, 2021

jyn514 commented on Oct 27, 2021

Mark-Simulacrum commented on Oct 27, 2021

jyn514 commented on Oct 27, 2021

Mark-Simulacrum commented on Oct 27, 2021

13 remaining items

petrochenkov commented on Aug 19, 2026

aerooneqq commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Non-deterministic overflow in `rustc_thread_pool` causing CI failures #90227