Skip to content

n2tiled-improved-scaling-on-multithread - #252

Merged
graeme-a-stewart merged 11 commits into
JuliaHEP:mainfrom
HarshitNagpal29:n2tiled-production-workspace
Sep 8, 2026
Merged

graeme-a-stewart merged 11 commits into
JuliaHEP:mainfrom
HarshitNagpal29:n2tiled-production-workspace

Conversation

@HarshitNagpal29

Copy link
Copy Markdown
Collaborator

Summary

This PR adds an opt-in reusable workspace for full N2Tiled reconstruction, aimed at improving multithreaded event-processing scalability by reducing per-event allocations and garbage-collection pressure.

Changes

  • Add N2TiledWorkspace for reusing jets, complete clustering history, tiling data, nearest-neighbour arrays, and inclusive-jet output.
  • Add callback-scoped with_n2tiled_reconstruction.
  • Add in-place inclusive_jets!.
  • Add explicit workspace-capacity release.
  • Preserve the existing owning jet_reconstruct and tiled_jet_reconstruct behaviour.
  • Add correctness, lifetime, concurrency, capacity-reuse, and allocation-regression tests.
  • Document workspace ownership and one-workspace-per-worker usage.

The workspace path still produces a complete ClusterSequence; users can perform normal history, constituent, exclusive-jet, and inclusive-jet queries inside the callback.

Compatibility

This is an additive API change. Existing users continue receiving independently owned results. High-throughput applications can opt into reusable storage and copy only the values that must be retained.

Validation

  • Workspace and owning paths produce exactly equal clustering results across multiple algorithms, radii, event sizes, and recombination schemes.
  • Warmed allocation was reduced from approximately 416 kB/event to 640 bytes/event in the local regression measurement.

@m-fila
m-fila requested review from graeme-a-stewart and m-fila and removed request for m-fila July 29, 2026 12:04
@codecov

codecov Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.78%. Comparing base (20868de) to head (ec1edda).

Files with missing lines Patch % Lines
src/ClusterSequence.jl 93.75% 1 Missing ⚠️
src/JetUtils.jl 95.65% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #252      +/-   ##
==========================================
+ Coverage   81.91%   82.78%   +0.86%     
==========================================
  Files          24       24              
  Lines        1521     1621     +100     
==========================================
+ Hits         1246     1342      +96     
- Misses        275      279       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@m-fila

m-fila commented Jul 29, 2026

Copy link
Copy Markdown
Member

In case you were using different version of formatter https://github.com/JuliaHEP/JetReconstruction.jl/blob/main/docs/src/contributing.md#formatting

@m-fila

m-fila commented Jul 29, 2026

Copy link
Copy Markdown
Member

I pushed to your branch to remove the unrelated formatting changes. Please pull to synchronize and avoid conflicts

@HarshitNagpal29

Copy link
Copy Markdown
Collaborator Author

I pushed to your branch to remove the unrelated formatting changes. Please pull to synchronize and avoid conflicts

Thank a lot sir.

Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLLStructs.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLLStructs.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/ClusterSequence.jl Resolved

@graeme-a-stewart graeme-a-stewart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HarshitNagpal29 - excellent work in designing something we can work with here.

I feel we have some way to go towards an optimal implementation with as few code path divergences in the implementation as possible (can we create a temporary workspace if there isn't a real one, then everything looks the same internally).

It also becomes more and more important to merge #247, as this PR is being broken every time we do a significant update of the code (currently it is broken by the fallout of #242). Having parameterised types is a big deal.

We also have to think about the convenient today of jet_reconstruct(particles; algorithm=..., which allows the user not to have to bother about the details of which reconstruction algorithm and strategy will be chosen. But here we start to move back to very explicit alg/strategy specific workspaces.

If we stick with this interface, we need to give q lot more examples of how to use the with_n2tiled_reconstruction for the most common cases.

And adding a multi-threaded example script in examples is essential.

Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLLStructs.jl Resolved
Comment thread src/ClusterSequence.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
Comment thread src/ClusterSequence.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
@graeme-a-stewart
graeme-a-stewart force-pushed the n2tiled-production-workspace branch from b51be3b to 5b598e1 Compare September 5, 2026 11:18

@graeme-a-stewart graeme-a-stewart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @HarshitNagpal29 - great, this is really converging. I think the code is good to go. I only have some comments about how we structure the documentation and a few of the docstrings.

Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread docs/src/strategy.md Resolved
Comment thread examples/n2tiled-multithreaded.jl Resolved
Comment thread src/ClusterSequence.jl Resolved
Comment thread src/JetReconstruction.jl Resolved
Comment thread src/TiledAlgoLL.jl Resolved
@graeme-a-stewart graeme-a-stewart added this to the 2.0.0 Release milestone Sep 5, 2026
HarshitNagpal29 and others added 3 commits September 7, 2026 17:55
Very minor documentation points.

Revert inclusive_jets() documentation.

@graeme-a-stewart graeme-a-stewart left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@graeme-a-stewart

Copy link
Copy Markdown
Member

@HarshitNagpal29 - I tweaked a few last documentation points (no point in ping-ponging them between us again).

Merging now - thanks so much for this! 👏

@graeme-a-stewart
graeme-a-stewart merged commit 0532a19 into JuliaHEP:main Sep 8, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants