Repository navigation
Conversation
…c supplied per generate call
… criteria and toy key threading
…lculator owns the fixed distribution; ToyCalculator requires its generator and key
… the Phi(-N) quantile of q_alt)
📝 WalkthroughWalkthroughThis PR refactors hypothesis testing around ChangesHypothesis testing API refactor
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/tips.md (1)
9-27: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the missing
AsymptoticCalculatorimportThe snippet uses
AsymptoticCalculatorbut only importsjaxandRootFindingLimitSolver, so it will raiseNameErroras written. Add:from everwillow.hypotest import AsymptoticCalculator🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/tips.md` around lines 9 - 27, Add the missing AsymptoticCalculator import in the documentation snippet, importing it from everwillow.hypotest alongside the existing jax and RootFindingLimitSolver imports so compute_limit can resolve the symbol.
🧹 Nitpick comments (1)
src/everwillow/_src/inference/hypotest/calculators.py (1)
537-570: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocstring says TypeError is conditional on a key being passed, but the check is unconditional.
upper_limit_bands'sRaisessection states: "TypeError: If a key is given with a solver that is not aStochasticLimitSolver." In_solve_limit(lines 474-498), theisinstance(solver, StochasticLimitSolver)check runs regardless of whetherkeywas passed (it defaults toself.keyeither way), so the TypeError fires even with nokeyargument.upper_limit's docstring at Line 529 phrases this correctly ("If the solver is not aStochasticLimitSolver"), so this looks like a copy/paste slip inupper_limit_bands.📝 Proposed docstring fix
Raises: - TypeError: If a key is given with a solver that is not a - `StochasticLimitSolver`. + TypeError: If the solver is not a `StochasticLimitSolver`. ValueError: If no solver is configured, or the criterion returns None.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/everwillow/_src/inference/hypotest/calculators.py` around lines 537 - 570, Correct the `upper_limit_bands` docstring’s `Raises` section to state that `TypeError` is raised whenever the solver is not a `StochasticLimitSolver`, without conditioning it on a supplied key. Keep the wording consistent with `upper_limit` and the unconditional validation in `_solve_limit`.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/tips.md`:
- Around line 9-27: Add the missing AsymptoticCalculator import in the
documentation snippet, importing it from everwillow.hypotest alongside the
existing jax and RootFindingLimitSolver imports so compute_limit can resolve the
symbol.
---
Nitpick comments:
In `@src/everwillow/_src/inference/hypotest/calculators.py`:
- Around line 537-570: Correct the `upper_limit_bands` docstring’s `Raises`
section to state that `TypeError` is raised whenever the solver is not a
`StochasticLimitSolver`, without conditioning it on a supplied key. Keep the
wording consistent with `upper_limit` and the unconditional validation in
`_solve_limit`.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8b957feb-892f-4cfb-8c4e-97472dc3a4df
📒 Files selected for processing (20)
docs/asimov-datasets.mddocs/extending-hypotest.mddocs/quickstart.mddocs/tips.mdpyproject.tomlsrc/everwillow/_src/inference/hypotest/calculators.pysrc/everwillow/_src/inference/hypotest/distributions.pysrc/everwillow/_src/inference/hypotest/limit_solvers.pysrc/everwillow/_src/inference/hypotest/results.pysrc/everwillow/_src/inference/hypotest/toys.pysrc/everwillow/_src/inference/hypotest/upper_limit.pysrc/everwillow/hypotest/__init__.pysrc/everwillow/hypotest/calculators.pysrc/everwillow/hypotest/limit_solvers.pysrc/everwillow/hypotest/upper_limit.pytests/inference/hypotest/test_calculators.pytests/inference/hypotest/test_distributions.pytests/inference/hypotest/test_limit_solvers.pytests/inference/hypotest/test_toys.pytests/inference/hypotest/test_upper_limit.py
💤 Files with no reviewable changes (3)
- src/everwillow/hypotest/upper_limit.py
- src/everwillow/_src/inference/hypotest/upper_limit.py
- tests/inference/hypotest/test_upper_limit.py
There was a problem hiding this comment.
Pull request overview
This PR refactors hypothesis-test upper-limit finding to use calculator methods driven by a pluggable LimitSolver contract, and makes toy-based upper limits a first-class workflow via ToyCalculator + keyed, stochastic-safe solvers. It also renames/standardizes expected-band computation to pvalue_bands and corrects the empirical Brazil-band quantile convention.
Changes:
- Introduces
LimitSolver(+RootFindingLimitSolver,GridScanLimitSolver,BisectionLimitSolver) and routesupper_limit()/upper_limit_bands()through calculators instead of free functions. - Adds
ToyCalculatorand updatesToyGeneratorto be a pure sampling engine with test statistic provided pergenerate()call; threads PRNG keys through toy workflows. - Renames
expected_pvalues→pvalue_bandsand fixes empirical expected-band quantile mapping (+N ↔ Φ(-N) quantile).
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/inference/hypotest/test_upper_limit.py | Removes tests for the deleted objective-based free functions. |
| tests/inference/hypotest/test_toys.py | Updates toy generator tests to pass test_statistic per generate() call and configure sampling via generator fields. |
| tests/inference/hypotest/test_limit_solvers.py | Adds contract/behavior tests for the new limit solver implementations, including pytree objectives and keyed reproducibility. |
| tests/inference/hypotest/test_distributions.py | Renames/adjusts tests to pvalue_bands and updates empirical band expectations to the corrected quantile convention. |
| tests/inference/hypotest/test_calculators.py | Major restructure: tests calculator base behavior, solver glue, criterion error contracts, fit_kwargs forwarding, ToyCalculator keyed toy regeneration, and result distribution recording. |
| src/everwillow/hypotest/upper_limit.py | Deletes the public re-export module for the removed free-function upper limit API. |
| src/everwillow/hypotest/limit_solvers.py | Adds a public re-export module for limit solvers. |
| src/everwillow/hypotest/calculators.py | Exposes ToyCalculator via the public calculators module. |
| src/everwillow/hypotest/init.py | Updates the top-level hypotest import example to reference the new solver-based API. |
| src/everwillow/_src/inference/hypotest/upper_limit.py | Deletes the internal implementation of objective-based upper-limit free functions. |
| src/everwillow/_src/inference/hypotest/toys.py | Refactors ToyGenerator to hold only sampling configuration and accept test_statistic per generate() call. |
| src/everwillow/_src/inference/hypotest/results.py | Adds distribution to HypoTestResult so downstream band computations use the producing distribution. |
| src/everwillow/_src/inference/hypotest/limit_solvers.py | Implements the new LimitSolver API and built-in solvers (root find, grid scan, bisection). |
| src/everwillow/_src/inference/hypotest/distributions.py | Renames expected_pvalues to pvalue_bands and fixes empirical expected-band quantile direction (+N ↔ Φ(-N)). |
| src/everwillow/_src/inference/hypotest/calculators.py | Refactors calculators into an abstract core with solver-driven limit methods; adds ToyCalculator and pvalue_bands() behavior based on result-carried distributions. |
| pyproject.toml | Adds a test-only lint ignore to allow _src imports by convention. |
| docs/tips.md | Updates the JIT tip to use calculator + RootFindingLimitSolver instead of free functions. |
| docs/quickstart.md | Updates expected bands API (pvalue_bands) and rewrites toy/limit docs around ToyCalculator + LimitSolver. |
| docs/extending-hypotest.md | Updates extension guidance to the new calculator/solver/toy generator contracts and pvalue_bands naming. |
| docs/asimov-datasets.md | Updates documentation to reference calc.pvalue_bands() instead of calc.expected(). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| A criterion is deterministic if evaluating it twice at the same POI gives the | ||
| same value. Criteria computed from asymptotic formulas behave this way. | ||
| Toy-based criteria do not: every evaluation throws a new set of toys, so the | ||
| returned CLs or p-value fluctuates with the statistical precision of the toy | ||
| ensemble. Solvers under this base are safe for toys because each evaluation is | ||
| used once, as a comparison against `level`, and never enters a fit or an | ||
| interpolation. Solvers outside this base may reuse or interpolate through | ||
| earlier evaluations, which is only valid for deterministic criteria. |
| # Update the search bracket based on the middle value | ||
| # narrow it: if mid_value is below level, lo moves to mid; if above, hi moves to mid | ||
| # This assumes the objective is montonically decreasing with increasing POIs |
| def upper_limit( | ||
| self, | ||
| solver: LimitSolver | None = None, | ||
| *, | ||
| level: float = 0.05, | ||
| criterion: tp.Callable[[HypoTestResult], PyTree] | None = None, | ||
| fit_kwargs: dict[str, tp.Any] | None = None, | ||
| ) -> Array: | ||
| """Find the upper limit: the POI value where the criterion crosses ``level``. |
| def upper_limit( | ||
| self, | ||
| solver: LimitSolver | None = None, | ||
| *, | ||
| level: float = 0.05, | ||
| criterion: tp.Callable[[HypoTestResult], PyTree] | None = None, | ||
| key: PRNGKeyArray | None = None, | ||
| fit_kwargs: dict[str, tp.Any] | None = None, | ||
| ) -> Array: |
| Attributes: | ||
| q_obs: Observed test statistic value. | ||
| pnull: p-value under the tested hypothesis (poi_test). | ||
| palt: p-value under the alternative hypothesis (poi_alt / background-only). | ||
| test_stat_result: Full test statistic result with fit information. | ||
| dist: The distribution of the test statistic which was used to compute p-values. | ||
| """ |
| Import from submodules directly:: | ||
|
|
||
| from everwillow.hypotest.test_statistics import QTilde | ||
| from everwillow.hypotest.calculators import AsymptoticCalculator | ||
| from everwillow.hypotest.distributions import QTildeAsymptotic | ||
| from everwillow.hypotest.upper_limit import upper_limit | ||
| from everwillow.hypotest.limit_solvers import RootFindingLimitSolver | ||
| """ |
pfackeldey
left a comment
There was a problem hiding this comment.
I think that looks good, if anything the Bisection Solver is kind of obsolete, because there's optimistix.Bisection already.
@pfackeldey the bisection in asymptotics uses the optimistix one, but the toys one is custom written to allow throwing toys in every iteration |
Restructures the hypotest calculators around a pluggable limit-solver contract and makes upper limits as class methods.
LimitSolvercontract with three implementations:RootFindingLimitSolver(optimistix root find, deterministic criteria),GridScanLimitSolver(fixed grid, observed limit and all bands in one pass),BisectionLimitSolver(stepped bisection with fresh toys per step). Solvers locate level crossings per leaf of any criterion pytree.StochasticLimitSolvermarks the algorithms that stay valid for noisy criteria;ToyCalculatoronly accepts solvers under that base.HypoTestCalculatorbecomes the abstract core (model binding,cls,pvalue_bands, limit methods).AsymptoticCalculatorowns the fixed distribution and the Asimov configuration.ToyCalculatorrequires atoy_generatorand akey, regenerates ensembles at every tested POI, and threads the key (field default with per-call override) throughtest,upper_limitandupper_limit_bands.upper_limit(solver=None, *, level, criterion, fit_kwargs)andupper_limit_bands(...)accept any user criterion; CLs is only the default. Results record the distribution that produced their p-values.ToyGeneratorreduced to a pure sampling engine; the test statistic is supplied pergeneratecall.Distribution.expected_pvaluesrenamed topvalue_bands.SimpleEmpiricalDistribution.pvalue_bandshad the Brazil band labels mirrored; band +N now maps to the Phi(-N) quantile of the background-only ensemble, matching the asymptotic convention and pyhf.Summary by CodeRabbit
pvalue_bandsacross hypothesis-testing workflows.