[qgallouedec]

Part of #7835, part of #7841.

GRPOTrainer logs rewards/<func>/mean as each function's mean over the completions it scored:

trl/trl/trainer/grpo_trainer.py

Lines 2960 to 2963 in 9507c44

	# Calculate mean reward per function, but only for samples where the function was applied (non-NaN values)
	for i, reward_func_name in enumerate(self.reward_func_names):
	mean_rewards = torch.nanmean(rewards_per_func[:, i]).item()
	self._metrics[mode][f"rewards/{reward_func_name}/mean"].append(mean_rewards)

AsyncGRPOTrainer averaged it over training samples instead, so it left out the groups dropped for zero reward std, and a rollout that forks into several samples counted once per sample. With the environment reward test's counters 1, 0, 0, that logs 0.44 where GRPO logs 1/3. It is now counted once per scored rollout, dropped groups included, like reward.

• Rollout traces no longer carry each function's reward per sample, since it is no longer a sample metric.

---

Note

Low Risk
Metric/logging semantics only; training loss and advantage logic are unchanged aside from removing duplicate per-sample reward means from traces.

Overview
AsyncGRPO now logs each rewards/<func>/mean the same way as GRPO: averaged over scored rollouts (including groups later dropped for zero reward std), not over training samples. Forked conversations that become several rows no longer inflate the mean, and dropped groups are included like the top-level reward metric.

The rollout worker accumulates per-function means in _rates during _score_group and stops attaching rewards/<func>/mean on individual RolloutSample metrics (per-sample group_std is unchanged). Trackio rollout traces therefore no longer carry per-function means on each sample. Docs and tests were updated to match.

^{Reviewed by Cursor Bugbot for commit 116041f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-10T22:10:26.101673Z	5faeb87	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

• en/async_grpo_trainer