[spike] cuda.compute host overhead investigation #11363

Activity

1. jrhemstad

   added this to CCCLon Sep 11, 2026
2. github-project-automation

   moved this to Todo in CCCLon Sep 11, 2026
3. jrhemstad

   assigned

   shwina

   on Sep 11, 2026
4. NaderAlAwar

   added a sub-issue on Sep 14, 2026

   • [cuda.compute]: Investigate making algorithm wrappers stateless to simplify multi threaded code #11388
5. shwina commented on Sep 21, 2026

   

   shwina

   on Sep 21, 2026

   Contributor

   More actions

   First PR addressing host overheads: #11497
6. leofang

   added

   cuda.computeFor all items related to the cuda.parallel Python moduleFor all items related to the cuda.parallel Python module

   on Sep 28, 2026
7. shwina commented on Sep 29, 2026

   

   shwina

   on Sep 29, 2026

   Contributor

   More actions

   Independent of any library change, callers can already avoid a large chunk of per-call overhead themselves:

   • Don't query temp-storage size on every call. The size only depends on shape/dtype, not on the call — query it once and reuse the buffer.
   • If your array sizes vary and you don't want to size for the worst case up front, track the largest size seen so far and only re-query when you see something bigger. (Query cost doesn't shrink for smaller sizes than what you've already sized for.)
   • Don't share one built algorithm object across threads without synchronization. These objects mutate shared state on every call; concurrent, unsynchronized calls into the same object can race.

   Follow-on issues for library-side fixes: #11732, #11733, #11734.
8. gevtushenko

   assigned

   oleksandr-pavlyk

   and unassigned

   shwina

   on Oct 3, 2026
9. shwina commented on Oct 6, 2026

   

   shwina

   on Oct 6, 2026

   Contributor

   More actions

   xref: #11895
10. jrhemstad

    added sub-issues on Oct 8, 2026

    • Bump the libcu++ ABI version after the cuda::cuda_error layout change #11886
    • [BUG]: [CUB][NVHPC] Mixed -gpu= fatbin: kernel bounds vs host launch CC mismatch #11572
11. jrhemstad

    removed a sub-issue on Oct 8, 2026

    • [BUG]: [CUB][NVHPC] Mixed -gpu= fatbin: kernel bounds vs host launch CC mismatch #11572
12. jrhemstad

    added sub-issues on Oct 8, 2026

    • Cache the validated stream handle by identity #11733
    • Extend per-call overhead fixes to scan, segmented reduce, and segmented sort #11734
13. jrhemstad

    removed a sub-issue on Oct 8, 2026

    • Bump the libcu++ ABI version after the cuda::cuda_error layout change #11886
14. jrhemstad

    added a sub-issue on Oct 8, 2026

    • Add an explicit-invalidation call path for reduce to skip redundant per-call work #11732
15. jrhemstad

    moved this from Todo to In Progress in CCCLon Oct 13, 2026
16. oleksandr-pavlyk

    added a sub-issue on Oct 22, 2026

    • Close performance gap between lookup of freshly built and deserialized algorithms #12161
17. oleksandr-pavlyk commented on Oct 28, 2026

    

    oleksandr-pavlyk

    on Oct 28, 2026

    Contributor

    More actions

    I think this PR needs to retain its original focus of doing work to profile host overhead: create scripts, collect and analyze data, create issues based on findings.

    Closing this issue does not require sub-issues to be closed (ref: https://github.com/orgs/community/discussions/154148#discussioncomment-13661487)
18. oleksandr-pavlyk commented on Nov 10, 2026

    

    oleksandr-pavlyk

    on Nov 10, 2026

    Contributor

    More actions

    I used cupyx.profiler.time_range context manager to insert NVTX ranges around steps of using cuda.compute. I did this at the script driving algorithm invokations, but also inside a method reimplementing steps performed in the __call__  method of each algorithm that was used to monkey patch the original implementation.  I then used nsys to collect profiling data. Here is what I see for reduce_into  for reduction of 2048 int32-valued vector (the size is deliberately chosen small to emphasize the overhead) with single level of NVTX ranges.  The workspace query takes 7.86 us, allocation of temp-storage takes 4.5 us, and call to enqueue actual computation takes 16.11 us.

    When I insert NVTX ranges at higher granularity, that introduces more overhead, but shows that only a small portion of workspace query overhead is actually spent in the call to CUB.

    These timelines do show significant time spent in prepare_initial_value step. Ashwin has a PR 12704 to allow user to avoid that step when appropriate (such as when initial value is known to developer to have the same value for all calls), or to avoid messing with state of stateless operators.