Skip to content

Rustdoc takes 8.5 GB of memory to document stm32h7 #79103

Closed
Closed
Rustdoc takes 8.5 GB of memory to document stm32h7#79103
Labels
C-bugCategory: This is a bug.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCI-compilememIssue: Problems and improvements with respect to memory usage during compilation.P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.T-rustdocRelevant to the rustdoc team, which will review and decide on the PR/issue.WG-embeddedWorking group: Embedded systems

Description

@jyn514
jyn514
opened on Nov 16, 2020

Similar to but not the same as #78761: rustdoc uses an inordinate amount of memory to document large crates.

heaptrack_gui shows almost a third of the memory being allocated in get_blanket_impls:

image

However, the peak usage is actually in check_mod_item_types, which rustdoc can't do much about:

image

Rustdoc should work improving on both the total memory usage and peak memory usage.

This is meant as a tracking issue, since there are issues other than get_blanket_impls causing high memory use, such as #76382. There is a Zulip stream for technical discussion.

cc @adamgreig

Activity

  1. jyn514
    added
    T-rustdocRelevant to the rustdoc team, which will review and decide on the PR/issue.
    I-compilememIssue: Problems and improvements with respect to memory usage during compilation.
    C-bugCategory: This is a bug.
    C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFC
    on Nov 16, 2020

  2. jyn514 commented on Nov 16, 2020

    @jyn514
    jyn514
    on Nov 16, 2020
    MemberAuthor

    This is causing docs.rs to have to periodically bump the sandbox limits when documenting stm32 crates: rust-lang/docs.rs#1179.

  3. jyn514
    added
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Nov 16, 2020
  4. camelid
    added
    P-mediumMedium priority
    and removed
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Dec 11, 2020

  5. camelid commented on Dec 11, 2020

    @camelid
    camelid
    on Dec 11, 2020
    Member

    P-medium: it's annoying that it takes so much memory, but it's not like you can't document it. And it's being worked on.

  6. jonas-schievink
    added
    WG-embeddedWorking group: Embedded systems
    on Dec 11, 2020

  7. jyn514 commented on Dec 11, 2020

    @jyn514
    jyn514
    on Dec 11, 2020
    MemberAuthor

    I ran Rustdoc under DHAT and found that a good tenth of the allocations come from get_blanket_impls alone:

    rust/src/librustdoc/clean/blanket_impl.rs

    Line 130 in 2225ee1

    items: self
    . I'm not sure how to upload the full 81 MB file to GitHub, but feel free to ping me here or on Zulip, I'll keep a copy around Here's a link to Google Drive with the full file. Unfortunately DHAT takes a full 50 minutes to run rustdoc on stm32 so it's a pain to collect on your own.

  8. jyn514 commented on Dec 11, 2020

    @jyn514
    jyn514
    on Dec 11, 2020
    MemberAuthor

    Another 7% just storing the structs themselves:

    rust/src/librustdoc/passes/collect_trait_impls.rs

    Line 128 in 2225ee1

    self.impls.extend(get_auto_trait_and_blanket_impls(
    . Maybe I should be asking just how many impls there are?

  9. 7 remaining items

  10. jyn514 commented on Dec 16, 2020

    @jyn514
    jyn514
    on Dec 16, 2020
    MemberAuthor

    Maybe I should be asking just how many impls there are?

    +-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-trait-impls                             | 33.31s    | 45.735          | 39.64s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.69s    | 32.537          | 23.69s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.81s     | 9.345           | 10.21s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.85s     | 3.913           | 4.01s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect-intra-doc-links                         | 918.47ms  | 1.261           | 970.03ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| expand_crate                                    | 905.65ms  | 1.244           | 909.95ms | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| build_extern_trait_impl                         | 796.51ms  | 1.094           | 1.38s    | 20080      |

    The strange thing that most of the time is not spent on build_extern_trait_impl, it's somewhere else in collect_trait_impls.

  11. jyn514 commented on Dec 16, 2020

    @jyn514
    jyn514
    on Dec 16, 2020
    MemberAuthor

    Almost all the time in collect_trait_impls is being spent on synthetic impls, specifically

    rust/src/librustdoc/passes/collect_trait_impls.rs

    Line 19 in 4031f7b

    let mut krate = synth.fold_crate(krate); 

    +-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| collect_synthetic_impls                         | 32.71s    | 44.967          | 37.56s   | 1          |

  12. jyn514 commented on Dec 16, 2020

    @jyn514
    jyn514
    on Dec 16, 2020
    MemberAuthor

    get_blanket_impls takes about 10x as long as get_synthetic_impls, but they're both pretty slow.

    +-------------------------------------------------+-----------+-----------------+----------+------------+
| Item                                            | Self time | % of total time | Time     | Item count |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_blanket_impls                               | 29.86s    | 41.053          | 34.48s   | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| render_html                                     | 23.18s    | 31.879          | 23.18s   | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| <unknown>                                       | 6.92s     | 9.514           | 10.33s   | 2417746    |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| get_auto_trait_impls                            | 3.42s     | 4.704           | 3.69s    | 32176      |
+-------------------------------------------------+-----------+-----------------+----------+------------+
| clean_crate                                     | 2.82s     | 3.871           | 3.98s    | 1          |
+-------------------------------------------------+-----------+-----------------+----------+------------+
  13. bors
    added 7 commits that reference this issue on Dec 16, 2020

    Auto merge of rust-lang#80041 - jyn514:shrink-item, r=GuillaumeGomez

    90f4b52

    Auto merge of rust-lang#80071 - jyn514:timings, r=Mark-Simulacrum

    353f3a3

    Auto merge of rust-lang#80095 - jyn514:stability-on-demand, r=Guillau…

    18b745e

    Auto merge of rust-lang#80099 - jyn514:visibility-on-demand, r=Guilla…

    28d73a3

    Auto merge of rust-lang#80014 - jyn514:box-item-kind, r=nnethercote

    158f8d0

  14. jyn514 commented on Jan 21, 2021

    @jyn514
    jyn514
    on Jan 21, 2021
    MemberAuthor

    This also impacts windows-rs.

  15. Manishearth
    added
    A-rustdoc
    and removed
    A-rustdoc
    on Apr 23, 2021

  16. jyn514 commented on Aug 22, 2021

    @jyn514
    jyn514
    on Aug 22, 2021
    MemberAuthor

    I profiled this recently, and it's down to 1.6 GB - still high, but a lot better than before. I'm going to close this until I find something more specific to track.

  17. jyn514
    closed this as completedon Aug 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCI-compilememIssue: Problems and improvements with respect to memory usage during compilation.P-mediumMedium priorityT-compilerRelevant to the compiler team, which will review and decide on the PR/issue.T-rustdocRelevant to the rustdoc team, which will review and decide on the PR/issue.WG-embeddedWorking group: Embedded systems

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions