Skip to content

Add CCCL agent skill with evals - #12579

Open
danielfrg wants to merge 2 commits into
NVIDIA:mainfrom
danielfrg:cccl-agent-skill
Open

danielfrg wants to merge 2 commits into
NVIDIA:mainfrom
danielfrg:cccl-agent-skill

Conversation

@danielfrg

Copy link
Copy Markdown
Contributor

Add skills/cccl for users of Thrust, CUB and libcu++.

Will be added to the NVIDIA skills catalog, with its evaluation dataset.

Description

closes

Results of evals in a 4090 (just for demonstration real eval numbers will run in the skills repo).

Without skill With skill
Assertions passed 209/243 (86.0%) 243/243 (100%)
Mean accuracy 0.776 0.944
Goal achieved 42/63 62/63
Skill loaded on positive cases 0/57 57/57
Skill loaded on negative cases 0/6 0/6
Mean tokens per run 149,634 111,827
Total cost $8.50 $7.39
Mean wall time per run 53.4 s 30.5 s
Median wall time per run 51.2 s 27.5 s

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Add skills/cccl, an agent skill for Thrust, CUB and libcu++ meant for the
NVIDIA skills catalog, with its evaluation dataset.

- SKILL.md: when to use CCCL instead of a hand-written kernel, fusing
  passes, silent wrong results, performance, execution environments,
  determinism, newer CCCL 3.x algorithms, block/warp primitives, libcu++,
  APIs removed in CCCL 3.0, and builds.
- references/libcudacxx.md: runtime API and parallel algorithms examples.
- evals/evals.json: 21 cases (19 positive, 2 negative) with starter code
  under evals/files/.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@danielfrg
danielfrg requested a review from a team as a code owner November 10, 2026 18:57
@danielfrg
danielfrg requested a review from miscco November 10, 2026 18:57
@github-project-automation github-project-automation Bot moved this to Todo in CCCL Nov 10, 2026
@copy-pr-bot

copy-pr-bot Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cccl-authenticator-app cccl-authenticator-app Bot moved this from Todo to In Review in CCCL Nov 10, 2026
@danielfrg
danielfrg requested review from gevtushenko and jrhemstad and removed request for miscco November 10, 2026 18:58
@danielfrg danielfrg self-assigned this Nov 10, 2026
@danielfrg danielfrg added the docs Documentation changes label Nov 10, 2026
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

  • Documentation
    • Added CUDA C++ guidance covering CCCL and libcu++ APIs, version compatibility, build workflows, algorithm selection, correctness, performance, determinism, and common pitfalls.
    • Added a libcu++ reference with toolkit guidance, usage constraints, and examples for streams and GPU parallel algorithms.
    • Added evaluation examples covering sorting, reductions, streams, reproducibility, CUDA version migration, and other development scenarios.

The pull request adds a CUDA C++ CCCL skill guide, a libcu++ reference, 21 evaluation cases, and sample programs covering CCCL builds, algorithms, correctness, performance, and execution patterns.

Changes: CCCL Skill and Evaluation Examples

Layer / File(s) Summary
CCCL guidance and references
skills/cccl/SKILL.md, skills/cccl/references/libcudacxx.md
The skill guide covers CCCL workflows, algorithms, correctness, performance, determinism, and execution patterns. The libcu++ reference documents APIs, examples, and usage constraints.
Evaluation prompts and assertions
skills/cccl/evals/evals.json
The evaluation records define prompts and expected-output assertions for CCCL tasks and out-of-scope questions.
Host-compiler build and statistics example
skills/cccl/evals/files/cccl-host-compiler-include/code/CMakeLists.txt, skills/cccl/evals/files/cccl-host-compiler-include/code/src/*
The sample project configures a C++/CUDA build, declares fleet-statistics interfaces, calculates GPU statistics, creates readings, and prints a report.
Thrust algorithms and execution examples
skills/cccl/evals/files/cccl-bykey-reproducible/code/ledger.cu, skills/cccl/evals/files/cccl-compose-events/code/replay.cu, skills/cccl/evals/files/cccl-fuse-passes/code/hot_particles.cu, skills/cccl/evals/files/cccl-sort-perf/code/bin_particles.cu, skills/cccl/evals/files/cccl-stream-pipeline/code/pipeline.cu
The programs demonstrate by-key reduction reproducibility, ordered update composition, particle diagnostics, sorting, and per-stream signal processing.
CUB reductions and ingest examples
skills/cccl/evals/files/cccl-port-2x-to-3x/code/stats.cu, skills/cccl/evals/files/cccl-review-ingest/code/ingest.cu
The programs use CUB reductions for calibrated readings and process sensor data with counts, offsets, radix sorting, and segmented minima.

Possibly related issues

  • NVIDIA/cccl#9466: The skill includes CCCL documentation guidance and links, which relates to the issue’s question about using developer documentation.
  • NVIDIA/cccl#9337: The pull request adds a CCCL skill and evaluation dataset, matching the epic’s request for CCCL skills.
  • NVIDIA/cccl#9995: The skill discusses determinism for per-key reductions and warns about thrust::reduce_by_key reproducibility for long key runs.

Possibly related PRs

  • NVIDIA/cccl#5583: This PR routes cub::DeviceReduce::ReduceByKey through the same API exercised by the new reproducibility evaluation.

Change: Feature

Merge Risk: Low · up to 19eb0

Users following the reference with CUDA 13.4 may encounter compilation failures. The version guidance needs correction.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: NVIDIA/cccl
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: 398c26d
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: 03a463d3-e124-4627-8391-feb99b4747cf
Commits

Reviewing files that changed from the base of the PR and between 04508f6 and 19eb098.

Files selected for processing: 1

  • skills/cccl/SKILL.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review info

Run configuration

  • Source mode: Auto
  • Repository: NVIDIA/cccl
  • Branch: main
  • File: .coderabbit.yaml
  • Commit: 398c26d
  • Layers: Repository YAML
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: dc2c79ce-7b97-4085-b72f-24e7b8facc13
Commits

Reviewing files that changed from the base of the PR and between 398c26d and 04508f6.

Files selected for processing: 15

  • skills/cccl/SKILL.md
  • skills/cccl/evals/evals.json
  • skills/cccl/evals/files/cccl-bykey-reproducible/code/ledger.cu
  • skills/cccl/evals/files/cccl-compose-events/code/replay.cu
  • skills/cccl/evals/files/cccl-fuse-passes/code/hot_particles.cu
  • skills/cccl/evals/files/cccl-host-compiler-include/code/CMakeLists.txt
  • skills/cccl/evals/files/cccl-host-compiler-include/code/src/gpu_stats.cu
  • skills/cccl/evals/files/cccl-host-compiler-include/code/src/gpu_stats.h
  • skills/cccl/evals/files/cccl-host-compiler-include/code/src/main.cpp
  • skills/cccl/evals/files/cccl-host-compiler-include/code/src/report.cpp
  • skills/cccl/evals/files/cccl-port-2x-to-3x/code/stats.cu
  • skills/cccl/evals/files/cccl-review-ingest/code/ingest.cu
  • skills/cccl/evals/files/cccl-sort-perf/code/bin_particles.cu
  • skills/cccl/evals/files/cccl-stream-pipeline/code/pipeline.cu
  • skills/cccl/references/libcudacxx.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@@ -0,0 +1,118 @@
# libcu++ beyond `cuda::std`

Everything here ships in the CUDA 13.4 toolkit (CCCL 3.5) under `<ctk>/include/cccl/cuda/`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Supported by static analysis

Script executed:

git rev-parse HEAD
git diff 5df91ca04257255ba13d3b744122c64a0c19d8a1 04508f60e2edf2054584271092c4bb4cc57d2bcf -- skills/cccl/references/libcudacxx.md
cat -n skills/cccl/references/libcudacxx.md
cat -n libcudacxx/include/cuda/std/__cccl/version.h | sed -n '1,100p'
rg -n -i 'cuda 13\.4|cccl 3\.5|13\.4.*3\.5|3\.5.*13\.4' -- README.md docs skills .github libcudacxx 2>/dev/null || test "$?" -eq 1

Repository: NVIDIA/cccl

Length of output: 29469


Web query:

CUDA Toolkit 13.4 CCCL 3.5 shipped version libcu++

Result:

Sources consulted:
- CUDA Toolkit 13.4 - Release Notes — Release Notes 13.4 documentation — https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html?highlight=ptx
- Releases · NVIDIA/cccl · GitHub — https://github.com/NVIDIA/cccl/releases
- GitHub - NVIDIA/cccl: CUDA Core Compute Libraries · GitHub — https://github.com/NVIDIA/cccl
- CUDA Toolkit 13.4 - Release Notes — Release Notes 13.4 documentation — https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html?from=20421
- CUDA Toolkit 13.4 Adds Windows on Arm Support and Greater Control over Shared GPUs | NVIDIA Technical Blog — https://developer.nvidia.com/blog?p=121255
- …and 21 more

Script executed:

cat -n docs/libcudacxx/extended_api/bit.rst | sed -n '1,80p'
cat -n docs/libcudacxx/extended_api/math.rst | sed -n '1,100p'
cat -n docs/libcudacxx/extended_api/warp.rst | sed -n '1,85p'
cat -n docs/libcudacxx/extended_api/numeric.rst | sed -n '1,105p'
cat -n docs/libcudacxx/standard_api/algorithms_library.rst | sed -n '1,35p'

Repository: NVIDIA/cccl

Length of output: 11833


suggestion: Correct the toolkit version and API availability.

CUDA 13.4 GA ships libcu++ 3.4.2. Code that uses cuda::warp_match_any can fail to compile with that toolkit.

Correct the version guidance

--- "a/skills/cccl/references/libcudacxx.md"
+++ "b/skills/cccl/references/libcudacxx.md"
@@ -1,6 +1,6 @@
 # libcu++ beyond `cuda::std`
 
-Everything here ships in the CUDA 13.4 toolkit (CCCL 3.5) under `<ctk>/include/cccl/cuda/`.
+CUDA 13.4 GA ships libcu++ 3.4.2 under `<ctk>/include/cccl/cuda/`. Some APIs require newer versions; `cuda::warp_match_any` requires CCCL 3.5 / CUDA 13.5.
 Older toolkits have less of it: check `CCCL_VERSION` and compile a small test first.
 
 ## Runtime API: streams, buffers, kernel launch
Committable suggestion

IMPORTANT
Review the replacement and indentation, then test before committing.

Suggested change
Everything here ships in the CUDA 13.4 toolkit (CCCL 3.5) under `<ctk>/include/cccl/cuda/`.
CUDA 13.4 GA ships libcu++ 3.4.2 under `<ctk>/include/cccl/cuda/`. Some APIs require newer versions; `cuda::warp_match_any` requires CCCL 3.5 / CUDA 13.5.

Comment thread skills/cccl/SKILL.md Resolved
@fbusato

fbusato commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

I would really love to see how it behaves with Programming Massively Parallel Processors examples

@NaderAlAwar

Copy link
Copy Markdown
Contributor

Results of evals in a 4090 (just for demonstration real eval numbers will run in the skills repo).

Where is the skills repo? Do they run evals to ensure that the skills don't regress performance?

@danielfrg

Copy link
Copy Markdown
Contributor Author

Results of evals in a 4090 (just for demonstration real eval numbers will run in the skills repo).

Where is the skills repo? Do they run evals to ensure that the skills don't regress performance?

https://github.com/nvidia/skills - i also sent you more info in slack

@danielfrg

Copy link
Copy Markdown
Contributor Author

I would really love to see how it behaves with Programming Massively Parallel Processors examples

thats a good idea! i dont think we want to add everything as evals (or maybe we do?) but we can get results for those!

@miscco miscco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Would have loved to see the PSTL algorithms but that is a wholly different topic

@danielfrg

Copy link
Copy Markdown
Contributor Author

I would really love to see how it behaves with Programming Massively Parallel Processors examples

It did pretty well when trying to complete the examples, for the ones where we have an algo: 6/12 went from hand written to using cccl, i think 12/12 would do it if it wasnt for some of the comments about only add compute, the agents were a bit to strict so they went back to hand written kernels.

@danielfrg danielfrg mentioned this pull request Nov 13, 2026
2 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation changes

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

4 participants