Repository navigation
Installing rust-docs component on Windows 10 is very slow #1540
Description
As reported on the User's Forum, installing the rust-docs component on Windows 10 is currently very slow compared to other components, even on machines with an SSD and multi-core processor.
I did some testing of Rustup on Windows with ProcessMonitor capturing all file events. The creation of a single file in rust-docs requires 18 “syscalls”:
8:56:27.9670280 AM rustup.exe 20964 CreateFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html NAME NOT FOUND Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a
8:56:27.9675751 AM rustup.exe 20964 CreateFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: OverwriteIf, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: 0, OpenResult: Created
8:56:27.9684873 AM rustup.exe 20964 QuerySecurityFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Information: Attribute
8:56:27.9685802 AM rustup.exe 20964 WriteFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Offset: 0, Length: 7,420, Priority: Normal
8:56:27.9687324 AM rustup.exe 20964 CloseFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS
8:56:27.9790328 AM rustup.exe 20964 CreateFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
8:56:27.9793741 AM rustup.exe 20964 QuerySecurityFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Information: Attribute
8:56:27.9794153 AM rustup.exe 20964 SetBasicInformationFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS CreationTime: 0, LastAccessTime: 2018-11-04 3:13:44 PM, LastWriteTime: 2018-11-04 3:13:44 PM, ChangeTime: 0, FileAttributes: n/a
8:56:27.9795474 AM rustup.exe 20964 CloseFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS
8:56:27.9799556 AM rustup.exe 20964 CreateFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Desired Access: Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
8:56:27.9800764 AM rustup.exe 20964 QuerySecurityFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Information: Attribute
8:56:27.9801085 AM rustup.exe 20964 QueryInformationVolume C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html BUFFER OVERFLOW VolumeCreationTime: 2017-01-18 7:26:52 PM, VolumeSerialNumber: 28BB-82BA, SupportsObjects: True, VolumeLabel: WinՂ
8:56:27.9801349 AM rustup.exe 20964 QueryAllInformationFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html BUFFER OVERFLOW CreationTime: 2018-11-05 8:56:27 AM, LastAccessTime: 2018-11-04 3:13:44 PM, LastWriteTime: 2018-11-04 3:13:44 PM, ChangeTime: 2018-11-05 8:56:27 AM, FileAttributes: A, AllocationSize: 8,192, EndOfFile: 7,420, NumberOfLinks: 1, DeletePending: False, Directory: False, IndexNumber: 0x8000000202cb1, EaSize: 0, Access: Read Attributes, Synchronize, Position: 0, Mode: Synchronous IO Non-Alert, AlignmentRequirement: Long
8:56:27.9801636 AM rustup.exe 20964 CloseFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS
8:56:27.9804884 AM rustup.exe 20964 CreateFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Desired Access: Write Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
8:56:27.9806919 AM rustup.exe 20964 QuerySecurityFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS Information: Attribute
8:56:27.9807270 AM rustup.exe 20964 SetBasicInformationFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS CreationTime: 0, LastAccessTime: 0, LastWriteTime: 0, ChangeTime: 0, FileAttributes: AN
8:56:27.9807689 AM rustup.exe 20964 CloseFile C:\Users\gordon.tyler\.rustup\tmp\tnchg8cf2r9owb5i_dir\rust-docs\share\doc\rust\html\std\macro.vec.html SUCCESS
Needing 5x CreateFile for each file definitely sounds bug-worthy…
Reacted by Geobert Quach, Martin Kavík, Vadim Petrochenkov, yks, Daniel Thul, Jacob Finkelman, Arsenii Lyashenko, ljedrz, Svend Tofte, Luo Jia / Zhouqi Jiang and 20 more
Activity
johnthagen commentedon Nov 9, 2018
CC @retep998
[-]Install rust-docs on Windows 10 is very slow[/-][+]Installing rust-docs on Windows 10 is very slow[/+][-]Installing rust-docs on Windows 10 is very slow[/-][+]Installing rust-docs component on Windows 10 is very slow[/+]johnthagen commentedon Nov 10, 2018
I did timed sample run:
All of the other components installed in a mater of a couple seconds, but
rust-docstook a staggering 2 minutes and 42 seconds to install! 😱This is on a Windows 10 machine with a quad core i5 @ 4GHz and an SSD. All 4 cores are pegged up at around 60-80% for the duration of the install.
johnthagen commentedon Nov 10, 2018
It appears that Windows Defender is definitely involved, based on a CPU usage capture during the install of
rust-docs:ndrewxie commentedon Nov 10, 2018
Yes, Windows Defender does cause significant slowdowns - however, turning off real time protection doesn't help very much - it still takes forever. Maybe I should be doing something more than just disabling real time protection? At this point I'm seriously annoyed - multiple people have opened issues for this, and suggested changes such as making the doc optional, merging the small files in the doc into a few large files to accommodate for slow file systems, etc - nothing has really been done so far except "disable your antivirus".
johnthagen commentedon Nov 10, 2018
@ndrewxie Could you please link those issues to this one? I opened this because I thought it hadn't been officially reported yet. Best to link any previous discussions together.
retep998 commentedon Nov 10, 2018
At this point I think the only fair thing to do is make
rust-docsinstallation slow on all other platforms too. That way the predominantly linux/mac using community of Rust will finally care about this problem and get it fixed.johnthagen commentedon Nov 10, 2018
As a polyglot Windows and Mac user, I do feel sometimes that Windows support lags behind Linux and Mac in subtle ways (like this issue, for example), but I'd like to try to keep this issue focused on this particular issue, and the concrete ways the situation can be improved.
Michael-F-Bryan commentedon Nov 13, 2018
I'm normally a Linux user, but found myself trying to do some project work on Windows today and my god is everything slow.
Are there concrete reasons why creating lots of small files (which
rustcloves doing) is such an expensive thing on Windows? I can see Windows Defender consuming quite a lot of CPU and disk IO so I'm assuming that has something to do with it.I'm not super familiar with how Windows does things like its filesystem and Windows Defender under the hood, but I'd be keen to help out. Besides telling all Windows users to turn off their antivirus are there any other ways to mitigate this poor performance? It sounds like we're not the only community suffering from poor filesystem performance because people have had the same issues with
node_modulesin the JavaScript world.kryptan commentedon Nov 15, 2018
It is also slow if you install Rust in WSL. Filesystem is just very slow on Windows. I would prefer to not install docs if it was possible.
johnthagen commentedon Nov 23, 2018
Another data point from @chriskrycho:
johnthagen commentedon Nov 29, 2018
More discussion on this internals thread.
Issue previously reported: #763
Issue to make rust-docs optional: #998
@nrc Appears to be working on profiles to help address this: #998 (comment)
doxxx commentedon Nov 30, 2018
The
rust-srccomponent is pretty slow to install on Windows as well. Not as bad as the docs, but it takes noticeably longer than any of the other components.doxxx commentedon Nov 30, 2018
I've been looking at the rustup code to see how it works with files. If I understand correctly, updating a component is performed in a transaction where each original file is moved to a temp folder before the new file is written in its place.
I was wondering if it would be viable to simply rename the toolchain folder? That would significantly reduce the number of file operations to perform. However, this would only work for toolchain updates, not component installation/removal.
That said, installing a new toolchain (i.e. not updating) is slow as well, so this may be a red herring.
40 remaining items
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
Allow more time for `rustup-init install`
kinnison commentedon Oct 18, 2019
With 1.20 you can set the profile to minimal and obviate the need to install
rust-docsso I consider this done. If you still want the docs and their installation remains too slow, please open a fresh issue to discuss possible mitigations.slonopotamus commentedon Oct 18, 2019
@kinnison: while I do agree that
minimalprofile allows to avoid installing documentation and resolves #998, the fact that documentation installation is slow on Windows is still valid.kinnison commentedon Oct 18, 2019
I agree that the form of rust-docs remains an issue, but it's not specific to Windows (though we are a lot better now providing you're not using something like McAfee) and any mitigations we come up with will need to be cross-platform or potentially something which changes how the rustdoc itself is generated.
uis246 commentedon Mar 24, 2024
I remember watching some video based on this issue about io performance and ms defender. Can anyone send link here please?
rami3l commentedon Mar 24, 2024
@uis246 Are you looking for https://lobste.rs/s/yne5kz/ntfs_really_isn_t_bad_robert_collins_lca?
uis246 commentedon Mar 24, 2024
@rami3l Yes, thank you.
LukeLIN-web commentedon Nov 17, 2024
I also meet this problem in linux,