Repository navigation
Getting file descriptor count on macOS throws #65254
Description
Description
A known way to get the file descriptor count on macOS is by using /dev/fd. Here is an example implementation in Rust.
The problem is that some file descriptors listed under /dev/fd are not stat-able, so they will fail when queried. One example is kqueues.
Reproduction Steps
Directory.EnumerateFileSystemEntries("/dev/fd").Count(); on macOS.
Expected behavior
The number of open file descriptors.
Actual behavior
C# > Directory.EnumerateFileSystemEntries("/dev/fd").Count();
╭─❌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ System.UnauthorizedAccessException: Access to the path '/dev/fd/5' is denied. │
│ ---> System.IO.IOException: Bad file descriptor │
│ --- End of inner exception stack trace --- │
│ at System.IO.FileStatus.ThrowOnCacheInitializationError(ReadOnlySpan`1 path) │
│ at System.IO.Enumeration.FileSystemEntry.get_IsSymbolicLink() │
│ at System.IO.Enumeration.FileSystemEntry.Initialize(FileSystemEntry& entry, DirectoryEntry directoryEntry, ReadOnlySpan`1 │
│ directory, ReadOnlySpan`1 rootDirectory, ReadOnlySpan`1 originalRootDirectory, Span`1 pathBuffer) │
│ at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext() │
│ at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source) │
│ at Submission#3.<<Initialize>>d__0.MoveNext() │
│ --- End of stack trace from previous location --- │
│ at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 │
│ currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken) │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Regression?
Not sure if it's a regression.
Known Workarounds
Manually PInvoke opendir and then iterate with readdir.
Configuration
dotnet 6.0.101, macOS 10.14+, x64 and arm64
Other information
No response
Activity
Therzok commentedon Feb 12, 2022
area-System.IO, I guess :Ddanmoseley commentedon Feb 13, 2022
@Therzok what do you propose here? Presumably in the general case, we do want to throw this exception.
Therzok commentedon Feb 13, 2022
The API implicitly does the evaluation of dir entries and their metadata. Something like directory entry count shouldn't populate the stat information of files, unless it's explicitly requested.
The equivalent C code is something like:
I'd understand if something like
GetFileSystemEntrieswould do implicit evaluation here to have a snapshot of the filesystem data, butEnumerateFileSystemEntriesis advertised as lazy. Shouldn't the entry properties be populated lazily?adamsitnik commentedon Feb 14, 2022
This code has been recently refactored by @tmds:
What we have in 6.0:
runtime/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEntry.Unix.cs
Lines 45 to 61 in 3d345e0
What we have in 7.0:
runtime/src/libraries/System.Private.CoreLib/src/System/IO/Enumeration/FileSystemEntry.Unix.cs
Lines 43 to 58 in e23382c
in theory it should not throw, let me start my Linux machine and check it
tmds commentedon Feb 14, 2022
Yes, this was refactored (and fixed) in #60214.
One thing that was missing is
continueOnError: trueforIsSymbolicLinkwhich is the cause for this exception.adamsitnik commentedon Feb 14, 2022
I've added a test that ensures that the bug is not coming back: #65301
kasperk81 commentedon Feb 14, 2022
is this fix planned for backporting to net6.0? if not, any workaround for users?
tmds commentedon Feb 14, 2022
Not using .NET APIs.
Maybe you can try PInvoking the
proc_pidinfoAPI mentioned in this article: https://zameermanji.com/blog/2021/8/1/counting-open-file-descriptors-on-macos/.adamsitnik commentedon Feb 14, 2022
I was not able to reproduce it using
net5.0, it's a regression that has been introduced innet6.0and it's specific to macOS.@tmds how difficult would it be to backport part of #60214 (the bug fix without refactor) to 6.0?
tmds commentedon Feb 15, 2022
It was introduced in #52235
Yes, and also specific to the special directory of
/dev/fdwhich represents the open file descriptors of the process.One of these file descriptors doesn't support
statprobably because its something special that is not ordinarily found in a real file system.#60214 is a bugfix PR, it doesn't do much refactoring. There are little things to be left out. I think you should take it as a whole.
There was a separate PR to refactor some things: #62721.
adamsitnik commentedon Feb 15, 2022
@tmds please excuse me, for some reason I mixed these two PRs
adamsitnik commentedon Feb 15, 2022
@kasperk81 in what circumstances have you hit this bug? Is it exactly the same use case?
adamsitnik commentedon Mar 14, 2022
I have not received any answers for a month, so I am going to close the issue (it's fixed in 7.0, we currently don't plan to backport it to 6.0)
Therzok commentedon Mar 14, 2022
None of the questions were directed towards me, so I'm pretty sure the issue should still be open, especially given it's a regression in net6.0
jozkee commentedon Apr 1, 2022
@danmoseley @jeffhandley does this meets the bar for backporting to 6.0?
I will check later if I can repro this on macOS.
jeffhandley commentedon Apr 6, 2022
@Therzok Could you describe the type of application this is impacting for you and whether you'd be able to adopt a preview release of .NET 7.0 to gain the fix? Additionally, it would be helpful if you could validate that the preview .NET 7 releases do in fact address the regression in your scenario. These data help us assess the risk/reward of backporting a fix.
Thanks!
Therzok commentedon Apr 7, 2022
Hey @jeffhandley , thanks for the response!
I work on Visual Studio for Mac. We use this API to check whether we are close to hitting the ridiculously low limit of open file handles (256). There currently is a workaround in place where we count filescriptors in C - did not have one at the time.
I'm not sure if publishing a release on top of .NET 7.0 is an option right now.
Testing on net7
Console.WriteLine(Directory.EnumerateFileSystemEntries("/dev/fd").Count());Yields
38which is correct and matcheslsof -p.lsof -p output
Closing as we added a workaround on our end, but having this API work in net6 would be nice.
jeffhandley commentedon Apr 7, 2022
Thank you, @Therzok. Since you have the (albeit unfortunate) workaround in place are are unblocked, and the backport isn't 100% straightforward/contained, I'm not inclined to backport the fix. If others report the same issue in a way that they are blocked, that would encourage the backport though.