Skip to content

feat: calculator upper limits through pluggable limit solvers with toy support - #66

Open
MoAly98 wants to merge 11 commits into
mainfrom
maly/improve-upper-limit
Open

MoAly98 wants to merge 11 commits into
mainfrom
maly/improve-upper-limit

Conversation

@MoAly98

@MoAly98 MoAly98 commented Jul 10, 2026 •

Copy link
Copy Markdown
Owner

Restructures the hypotest calculators around a pluggable limit-solver contract and makes upper limits as class methods.

  • LimitSolver contract with three implementations: RootFindingLimitSolver (optimistix root find, deterministic criteria), GridScanLimitSolver (fixed grid, observed limit and all bands in one pass), BisectionLimitSolver (stepped bisection with fresh toys per step). Solvers locate level crossings per leaf of any criterion pytree. StochasticLimitSolver marks the algorithms that stay valid for noisy criteria; ToyCalculator only accepts solvers under that base.
  • HypoTestCalculator becomes the abstract core (model binding, cls, pvalue_bands, limit methods). AsymptoticCalculator owns the fixed distribution and the Asimov configuration. ToyCalculator requires a toy_generator and a key, regenerates ensembles at every tested POI, and threads the key (field default with per-call override) through test, upper_limit and upper_limit_bands.
  • upper_limit(solver=None, *, level, criterion, fit_kwargs) and upper_limit_bands(...) accept any user criterion; CLs is only the default. Results record the distribution that produced their p-values.
  • ToyGenerator reduced to a pure sampling engine; the test statistic is supplied per generate call. Distribution.expected_pvalues renamed to pvalue_bands.
  • Breaking: the objective-based upper limit free functions are removed in favour of the calculator methods.
  • Fix: SimpleEmpiricalDistribution.pvalue_bands had the Brazil band labels mirrored; band +N now maps to the Phi(-N) quantile of the background-only ensemble, matching the asymptotic convention and pyhf.
  • The toy pipeline was validated against pyhf (test statistics to 1e-7, ensembles, CLs values, bands and limits) and against exact Poisson enumeration on counting models. Follow-up improvements are tracked in Toy-based CLs: potential improvements and features #65.
  • Docs updated to the calculator limit API throughout.

Summary by CodeRabbit

  • New Features
    • Added configurable limit solvers for root-finding, grid scans, and stochastic bisection.
    • Added support for toy-based upper limits with reproducible random-key handling.
    • Exposed ToyCalculator and limit-solving tools through the public API.
  • Improvements
    • Renamed expected p-value calculations to pvalue_bands across hypothesis-testing workflows.
    • Updated toy generation and calculator workflows for clearer configuration.
  • Documentation
    • Updated quickstart, extensibility, dataset, and JIT guidance to reflect the new APIs and limit-solving model.

@coderabbitai

coderabbitai Bot commented Jul 10, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR refactors hypothesis testing around pvalue_bands, result-owned distributions, pluggable deterministic or stochastic limit solvers, and per-call toy test statistics. It updates calculators, exports, tests, and documentation to reflect the revised APIs.

Changes

Hypothesis testing API refactor

Layer / File(s) Summary
Distribution and result contracts
src/everwillow/_src/inference/hypotest/results.py, src/everwillow/_src/inference/hypotest/distributions.py, tests/inference/hypotest/test_distributions.py
Expected-pvalue APIs are renamed to pvalue_bands, distributions are stored on HypoTestResult, and empirical band quantiles are updated.
Calculator and toy execution flow
src/everwillow/_src/inference/hypotest/calculators.py, src/everwillow/_src/inference/hypotest/toys.py, tests/inference/hypotest/test_calculators.py, tests/inference/hypotest/test_toys.py
Calculators populate result distributions, ToyGenerator receives test statistics per generation call, and toy-based limit evaluations thread PRNG keys.
Pluggable limit solving
src/everwillow/_src/inference/hypotest/limit_solvers.py, src/everwillow/hypotest/limit_solvers.py, tests/inference/hypotest/test_limit_solvers.py, docs/quickstart.md, docs/extending-hypotest.md
Adds root-finding, grid-scan, and bisection solver APIs and integrates them with calculator upper-limit methods and updated documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

  • MoAly98/everwillow#35: Introduced the upper-limit utilities that this PR replaces with LimitSolver implementations.
  • MoAly98/everwillow#45: Updated related test-statistic and ToyGenerator wiring extended by this refactor.

Suggested reviewers: pfackeldey

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 67.97% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: calculator upper limits via pluggable limit solvers with toy support.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch maly/improve-upper-limit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/tips.md (1)

9-27: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add the missing AsymptoticCalculator import

The snippet uses AsymptoticCalculator but only imports jax and RootFindingLimitSolver, so it will raise NameError as written. Add:

from everwillow.hypotest import AsymptoticCalculator
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/tips.md` around lines 9 - 27, Add the missing AsymptoticCalculator
import in the documentation snippet, importing it from everwillow.hypotest
alongside the existing jax and RootFindingLimitSolver imports so compute_limit
can resolve the symbol.
🧹 Nitpick comments (1)
src/everwillow/_src/inference/hypotest/calculators.py (1)

537-570: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Docstring says TypeError is conditional on a key being passed, but the check is unconditional.

upper_limit_bands's Raises section states: "TypeError: If a key is given with a solver that is not a StochasticLimitSolver." In _solve_limit (lines 474-498), the isinstance(solver, StochasticLimitSolver) check runs regardless of whether key was passed (it defaults to self.key either way), so the TypeError fires even with no key argument. upper_limit's docstring at Line 529 phrases this correctly ("If the solver is not a StochasticLimitSolver"), so this looks like a copy/paste slip in upper_limit_bands.

📝 Proposed docstring fix
         Raises:
-            TypeError: If a key is given with a solver that is not a
-                `StochasticLimitSolver`.
+            TypeError: If the solver is not a `StochasticLimitSolver`.
             ValueError: If no solver is configured, or the criterion returns
                 None.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/everwillow/_src/inference/hypotest/calculators.py` around lines 537 -
570, Correct the `upper_limit_bands` docstring’s `Raises` section to state that
`TypeError` is raised whenever the solver is not a `StochasticLimitSolver`,
without conditioning it on a supplied key. Keep the wording consistent with
`upper_limit` and the unconditional validation in `_solve_limit`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/tips.md`:
- Around line 9-27: Add the missing AsymptoticCalculator import in the
documentation snippet, importing it from everwillow.hypotest alongside the
existing jax and RootFindingLimitSolver imports so compute_limit can resolve the
symbol.

---

Nitpick comments:
In `@src/everwillow/_src/inference/hypotest/calculators.py`:
- Around line 537-570: Correct the `upper_limit_bands` docstring’s `Raises`
section to state that `TypeError` is raised whenever the solver is not a
`StochasticLimitSolver`, without conditioning it on a supplied key. Keep the
wording consistent with `upper_limit` and the unconditional validation in
`_solve_limit`.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8b957feb-892f-4cfb-8c4e-97472dc3a4df

📥 Commits

Reviewing files that changed from the base of the PR and between 528dd62 and 11c68e3.

📒 Files selected for processing (20)
  • docs/asimov-datasets.md
  • docs/extending-hypotest.md
  • docs/quickstart.md
  • docs/tips.md
  • pyproject.toml
  • src/everwillow/_src/inference/hypotest/calculators.py
  • src/everwillow/_src/inference/hypotest/distributions.py
  • src/everwillow/_src/inference/hypotest/limit_solvers.py
  • src/everwillow/_src/inference/hypotest/results.py
  • src/everwillow/_src/inference/hypotest/toys.py
  • src/everwillow/_src/inference/hypotest/upper_limit.py
  • src/everwillow/hypotest/__init__.py
  • src/everwillow/hypotest/calculators.py
  • src/everwillow/hypotest/limit_solvers.py
  • src/everwillow/hypotest/upper_limit.py
  • tests/inference/hypotest/test_calculators.py
  • tests/inference/hypotest/test_distributions.py
  • tests/inference/hypotest/test_limit_solvers.py
  • tests/inference/hypotest/test_toys.py
  • tests/inference/hypotest/test_upper_limit.py
💤 Files with no reviewable changes (3)
  • src/everwillow/hypotest/upper_limit.py
  • src/everwillow/_src/inference/hypotest/upper_limit.py
  • tests/inference/hypotest/test_upper_limit.py

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors hypothesis-test upper-limit finding to use calculator methods driven by a pluggable LimitSolver contract, and makes toy-based upper limits a first-class workflow via ToyCalculator + keyed, stochastic-safe solvers. It also renames/standardizes expected-band computation to pvalue_bands and corrects the empirical Brazil-band quantile convention.

Changes:

  • Introduces LimitSolver (+ RootFindingLimitSolver, GridScanLimitSolver, BisectionLimitSolver) and routes upper_limit() / upper_limit_bands() through calculators instead of free functions.
  • Adds ToyCalculator and updates ToyGenerator to be a pure sampling engine with test statistic provided per generate() call; threads PRNG keys through toy workflows.
  • Renames expected_pvalues → pvalue_bands and fixes empirical expected-band quantile mapping (+N ↔ Φ(-N) quantile).

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tests/inference/hypotest/test_upper_limit.py Removes tests for the deleted objective-based free functions.
tests/inference/hypotest/test_toys.py Updates toy generator tests to pass test_statistic per generate() call and configure sampling via generator fields.
tests/inference/hypotest/test_limit_solvers.py Adds contract/behavior tests for the new limit solver implementations, including pytree objectives and keyed reproducibility.
tests/inference/hypotest/test_distributions.py Renames/adjusts tests to pvalue_bands and updates empirical band expectations to the corrected quantile convention.
tests/inference/hypotest/test_calculators.py Major restructure: tests calculator base behavior, solver glue, criterion error contracts, fit_kwargs forwarding, ToyCalculator keyed toy regeneration, and result distribution recording.
src/everwillow/hypotest/upper_limit.py Deletes the public re-export module for the removed free-function upper limit API.
src/everwillow/hypotest/limit_solvers.py Adds a public re-export module for limit solvers.
src/everwillow/hypotest/calculators.py Exposes ToyCalculator via the public calculators module.
src/everwillow/hypotest/init.py Updates the top-level hypotest import example to reference the new solver-based API.
src/everwillow/_src/inference/hypotest/upper_limit.py Deletes the internal implementation of objective-based upper-limit free functions.
src/everwillow/_src/inference/hypotest/toys.py Refactors ToyGenerator to hold only sampling configuration and accept test_statistic per generate() call.
src/everwillow/_src/inference/hypotest/results.py Adds distribution to HypoTestResult so downstream band computations use the producing distribution.
src/everwillow/_src/inference/hypotest/limit_solvers.py Implements the new LimitSolver API and built-in solvers (root find, grid scan, bisection).
src/everwillow/_src/inference/hypotest/distributions.py Renames expected_pvalues to pvalue_bands and fixes empirical expected-band quantile direction (+N ↔ Φ(-N)).
src/everwillow/_src/inference/hypotest/calculators.py Refactors calculators into an abstract core with solver-driven limit methods; adds ToyCalculator and pvalue_bands() behavior based on result-carried distributions.
pyproject.toml Adds a test-only lint ignore to allow _src imports by convention.
docs/tips.md Updates the JIT tip to use calculator + RootFindingLimitSolver instead of free functions.
docs/quickstart.md Updates expected bands API (pvalue_bands) and rewrites toy/limit docs around ToyCalculator + LimitSolver.
docs/extending-hypotest.md Updates extension guidance to the new calculator/solver/toy generator contracts and pvalue_bands naming.
docs/asimov-datasets.md Updates documentation to reference calc.pvalue_bands() instead of calc.expected().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +63 to +70
A criterion is deterministic if evaluating it twice at the same POI gives the
same value. Criteria computed from asymptotic formulas behave this way.
Toy-based criteria do not: every evaluation throws a new set of toys, so the
returned CLs or p-value fluctuates with the statistical precision of the toy
ensemble. Solvers under this base are safe for toys because each evaluation is
used once, as a comparison against `level`, and never enters a fit or an
interpolation. Solvers outside this base may reuse or interpolate through
earlier evaluations, which is only valid for deterministic criteria.
Comment on lines +285 to +287
# Update the search bracket based on the middle value
# narrow it: if mid_value is below level, lo moves to mid; if above, hi moves to mid
# This assumes the objective is montonically decreasing with increasing POIs
Comment on lines +182 to +190
def upper_limit(
self,
solver: LimitSolver | None = None,
*,
level: float = 0.05,
criterion: tp.Callable[[HypoTestResult], PyTree] | None = None,
fit_kwargs: dict[str, tp.Any] | None = None,
) -> Array:
"""Find the upper limit: the POI value where the criterion crosses ``level``.
Comment on lines +500 to +508
def upper_limit(
self,
solver: LimitSolver | None = None,
*,
level: float = 0.05,
criterion: tp.Callable[[HypoTestResult], PyTree] | None = None,
key: PRNGKeyArray | None = None,
fit_kwargs: dict[str, tp.Any] | None = None,
) -> Array:
Comment on lines 123 to 129
Attributes:
q_obs: Observed test statistic value.
pnull: p-value under the tested hypothesis (poi_test).
palt: p-value under the alternative hypothesis (poi_alt / background-only).
test_stat_result: Full test statistic result with fit information.
dist: The distribution of the test statistic which was used to compute p-values.
"""
Comment on lines 3 to 9
Import from submodules directly::

from everwillow.hypotest.test_statistics import QTilde
from everwillow.hypotest.calculators import AsymptoticCalculator
from everwillow.hypotest.distributions import QTildeAsymptotic
from everwillow.hypotest.upper_limit import upper_limit
from everwillow.hypotest.limit_solvers import RootFindingLimitSolver
"""
@MoAly98
MoAly98 requested a review from pfackeldey July 22, 2026 13:10

@pfackeldey pfackeldey left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that looks good, if anything the Bisection Solver is kind of obsolete, because there's optimistix.Bisection already.

@MoAly98

MoAly98 commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

I think that looks good, if anything the Bisection Solver is kind of obsolete, because there's optimistix.Bisection already.

@pfackeldey the bisection in asymptotics uses the optimistix one, but the toys one is custom written to allow throwing toys in every iteration

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants