[cswjl]

What does this PR do?

Adds the Unbiased Reward Model (URM) loss to RewardTrainer as loss_type="urm", from our ICML 2026 paper Unbiased Alignment for Large Language Models with Noisy Preferences. This is the RewardTrainer half of #8175, split out as requested there. The DPOTrainer half (UDPO) is in #8211.

URM corrects the Bradley-Terry loss for preference labels flipped with probability ε. This PR implements its normalized form, called α-URM in the paper:

$$ \mathcal{L}_{\text{URM}} = -\alpha \log\frac{e^{\Delta} + a}{e^{\Delta} + 1}, \qquad \Delta = r(x, y_w) - r(x, y_l), \qquad a = \frac{\varepsilon}{1 - \varepsilon}, \qquad \alpha = \frac{1 + \sqrt{a}}{1 - \sqrt{a}} $$

The constant α keeps the maximum gradient magnitude at 1, as for the Bradley-Terry loss. This is the variant that the paper recommends to practitioners; the unnormalized loss (Eq. 10) only differs by this constant.

from trl import RewardConfig

training_args = RewardConfig(loss_type="urm", label_smoothing=0.4)

Changes:

• RewardConfig: new loss_type ("sigmoid" by default, or "urm") and label_smoothing fields. label_smoothing is ε, as loss_type="robust" uses it in DPOConfig.
• RewardTrainer: the log term is computed in log space as logaddexp(logsigmoid(Δ), log(a) + logsigmoid(-Δ)) for numerical stability, and label_smoothing=0.0 recovers "sigmoid". The default loss is unchanged: the margin is now subtracted before the loss branch, which computes the same expression as before. A ValueError is raised when label_smoothing is outside [0.0, 0.5), because a ≥ 1 no longer prefers the chosen completions.
• Docs: a "Loss types" table in reward_trainer.md and a paper_index.md entry.
• Tests: training with the loss, the loss value against the closed form (single eval example, compared with the logged reward margin), and the label_smoothing range check.

Part of #8175

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

@qgallouedec, as requested in #8175.

---

Note

Medium Risk
Changes reward-model training loss paths and adds hyperparameter validation; default sigmoid training is preserved but new options affect alignment-critical reward training.

Overview
Adds URM (α-URM) noisy-preference training to RewardTrainer via new RewardConfig options loss_type="urm" and label_smoothing (flip probability ε).

The trainer now selects between the existing Bradley-Terry ("sigmoid") objective and the normalized URM loss, with log-space stabilization, margin handling shared across both, and a ValueError when ε is outside [0.0, 0.5). Default behavior stays "sigmoid" with label_smoothing=0.0.

Documentation covers the loss table and paper recipe in reward_trainer.md and paper_index.md. Tests cover URM training, closed-form eval loss, and invalid label_smoothing.

^{Reviewed by Cursor Bugbot for commit 49778c9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.