Tracking issue for fs::walk_dir #27707

Activity

1. alexcrichton

   added

   A-io

   T-libs-api[DEPRECATED; DO NOT USE][DEPRECATED; DO NOT USE]

   B-unstableBlocker: Implemented in the nightly compiler and unstable.Blocker: Implemented in the nightly compiler and unstable.

   on Aug 12, 2015
2. sfackler commented on Aug 12, 2015

   

   sfackler

   on Aug 12, 2015

   Member

   More actions

   I've found Java's Files::walk_file_tree really useful since it supports things like stopping iteration early and running code at different points in the iteration.

   https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#walkFileTree-java.nio.file.Path-java.nio.file.FileVisitor-
3. BurntSushi commented on Sep 17, 2015

   

   BurntSushi

   on Sep 17, 2015

   Member

   More actions

   I'd like to take the conn on this, and echo this thought:

   Note that this is a large enough feature that this may want to be prototyped outside the standard library first and then perhaps consider moving in at a later date.

   I've taken a brief survey other other implementations which range from every-feature-in-the-book (ftw in glibc) to very minimal (Go's filepath.Walk), so there's a lot of variation on what we can do. I'd like to start with the following goals:

   • Fix the "too many open fds" issue. I am particularly fond of making the common case fast and falling back to some kind of memory allocation if we exceed some preset (but configurable) limit on the number of open file descriptors. (I think this probably means switching to depth first search, since it "feels" like shallow but bigger directories are more common than very deeply nested directories. In particular, with depth first search, we can guarantee that at most one file descriptor is open for each depth.)
   • Provide some kind of facility for traversing symlinks (but this should be configurable and will require extra resources if we want to preserve the invariant that every item is yielded exactly once).
   • Provide control over iteration (e.g., "don't descend into this directory" or "skip the rest of this directory.")

   Most of the interfaces I've seen use a user-provided function to control iteration or even response to errors. I feel like we should probably pursue the same functionality but with iterators if possible, although I'm not sure what it will look like yet. Possibly methods on the iterator itself? (Although this is a design that is not too common in the Rust world.)

   How does this sound for a reasonable starting point?

   👍React with 👍1Reacted by Joshua Koo
4. SimonSapin commented on Sep 17, 2015

   

   SimonSapin

   on Sep 17, 2015

   Contributor

   More actions

   This may also be worth looking at:

   What’s New In Python 3.5: PEP 471 - os.scandir() function – a better and faster directory iterator
5. huonw commented on Sep 17, 2015

   

   huonw

   on Sep 17, 2015

   Contributor

   More actions

   @SimonSapin fwiw, that PEP has apparently already been taken into account for the current design: #16236.
6. SimonSapin commented on Sep 17, 2015

   

   SimonSapin

   on Sep 17, 2015

   Contributor

   More actions

   Never mind, then :)
7. alexcrichton commented on Sep 17, 2015

   

   alexcrichton

   on Sep 17, 2015

   MemberAuthor

   More actions

   @BurntSushi that all sounds great to me! I know that whenever I've wanted to use a fs walk-like function I've always wanted the option to say "don't descend into this directory" so it's certainly something I'd like to see!

   This definitely seems like a good candidate for development outside the standard library as it can be a relatively meaty piece of functionality and in theory our fs bindings should give you all the support you need to build this.
8. BurntSushi commented on Sep 27, 2015

   

   BurntSushi

   on Sep 27, 2015

   Member

   More actions

   I've finished my initial implementation: http://burntsushi.net/rustdoc/walkdir/

   Overall, I'm pretty happy with it. It isn't clear to me whether it's destined for std. In particular, the crate's custom DirEntry type has grown a fair bit of complexity, mostly due to the handling of following symbolic links. Should I just let it bake on crates.io for now? Write an RFC? (I have acquired a fair amount of institutional knowledge that might be good to encode into an RFC, even if the intention is to postpone.) Do we deprecate std::fs::walk_dir?
9. alexcrichton commented on Sep 28, 2015

   

   alexcrichton

   on Sep 28, 2015

   MemberAuthor

   More actions

   I think the best path forward would be to maybe let it bake for a bit, move it to the nursery, see how it turns out, and then eventually write an RFC to become an official rust-lang crate, and perhaps finally move it into the standard library. I don't necessarily feel an urgency to move it into the standard library right now, and the fast iteration outside seems quite useful for now!
10. BurntSushi commented on Sep 28, 2015

    

    BurntSushi

    on Sep 28, 2015

    Member

    More actions

    Sounds good to me!
11. steveklabnik commented on Sep 28, 2015

    

    steveklabnik

    on Sep 28, 2015

    Contributor

    More actions

    👍

    On Mon, Sep 28, 2015 at 6:56 AM, Andrew Gallant notifications@github.com
    wrote:

    Sounds good to me!

    —
    Reply to this email directly or view it on GitHub
    #27707 (comment).
12. aturon commented on Nov 3, 2015

    

    aturon

    on Nov 3, 2015

    Contributor

    More actions

    I wonder if we should go ahead and deprecate the in-tree version? Is there a path toward stabilization as-is?

    Nominating for 1.6 discussion.
13. aturon

    added

    I-nominated

    on Nov 3, 2015
14. 7 remaining items

    Load more
15. alexcrichton

    added a commit that references this issue on Dec 5, 2015

    std: Stabilize APIs for the 1.6 release

    ...

    This issue will close once commit 464cdff is merged into the 'incoming' branch.

    464cdff
16. bors

    closed this as completedin #30187on Dec 6, 2015