[solarsys]

Stacked on #1319, then #1318, #1317 and #1316. Review the last commit, autoimprove: feature search.

Why

#1319 lets you write and check features by hand. This PR lets the improver search for them. Searching features as well as settings brings three risks:

• noise features that win by chance;
• redundant features that pile up;
• features that work for reasons nobody can defend.

Each part below addresses one of these.

What changes

• A trial is a config plus a feature set. trials.jsonl records the full spec list, the group the trial added, whether it was accepted, and any suspicion.

• Alternating cycles (feature_cycles=(2, 1)):

  • Feature cycles hold the model at the current selection, and each agent proposes a group of specs with rationale and expected direction.
  • Model cycles hold the accepted features and propose configs.

  With one side fixed, every accepted change has a gain of its own.

• Forward selection. A group joins the accepted set only if its paired gain over the accepted set clears the lower bound. Specs that fail the future-blind test are refused before training, and the reason goes back to the proposer. A rejected group never becomes the result.

• Proposers:

  • LLMProposer.propose_features sees the operator menu, the task card, the accepted features and the earlier groups.
  • RandomProposer proposes random valid specs, labelled as having no rationale.
  • New QueueProposer tries a ranked list in order, with no LLM.

• confirm() (confirm=True). The leading candidates are re-scored on new folds with 3 training seeds, and must clear a stricter bound, confirm_se=2. Candidates picked as the best of many on the same folds are partly lucky.

• prune(). Accepted groups whose removal costs less than one SE are dropped, and the pruned set is the result. The ablation records what each group added.

• Safeguards:

  • Suspicion check: a new feature with ROC-AUC above 0.95 on its own is held as possible leakage.
  • Review gate: features with no rationale, an opposite learned direction, or a suspicion are held back until approved. The test then also scores the model without them.
  • early_stop: a trial that is clearly worse is stopped after 2 folds.

• The report gains Features (each feature's definition, rationale, expected direction, contribution and review status) and Confirmation (confirmed scores, the ablation and what was pruned).

What the tests showed, and what changed because of it

The planted-data tests found two statistical weaknesses. Both are fixed here:

1. A noise feature, and an exact duplicate of it, passed the lenient one-SE acceptance bound by chance. In a small world the paired SE is noisy; the duplicate's "gain" was pure optimization noise. Confirmation and pruning now remove them. The test asserts the end state, not each provisional acceptance.
2. With shuffled labels, a feature group survived. It had been rejected during the search, yet it came back as a candidate in confirmation and won by chance (+0.046 ± 0.032). Rejected groups are now never candidates, and the confirmed selection must clear 2 SE.

Tests

tests/core/test_autoimprove_search.py covers the proposal's done-when list with 6 tests on a synthetic event log, where the outcome depends on recent visits, with 15% label noise:

• Planted feature: found, and the ablation attributes its gain to it (+0.28 PR-AUC). The "any visit" flag, which it makes redundant, is pruned. The test gain over baseline is above 0.05.
• Shuffled labels: nothing survives confirmation; the baseline stays.
• Reversed effect: a planted feature whose stated direction is wrong is flagged, held back, scored without, then released on approval.
• Leakage: a feature that encodes the outcome is held as suspicious and not accepted.
• LLM path: features proposed from the menu; the hypothesis is recorded.
• Resume and early stop: a run resumes from trials.jsonl, and early stop works.

All 49 autoimprove tests pass, as do the doctests, ruff and tools/check_pr_rules.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ntMLEmFsQEQJKTmPGGw4T

[github-actions]

Test Coverage

Total: 70.76% (19698 / 27838 lines)

Modified file	Coverage	Lines
pyhealth/autoimprove/__init__.py	100.00%	8 / 8
pyhealth/autoimprove/evaluate.py	92.57%	162 / 175
pyhealth/autoimprove/explain.py	89.16%	181 / 203
pyhealth/autoimprove/features.py	88.36%	524 / 593
pyhealth/autoimprove/improver.py	91.67%	792 / 864
pyhealth/autoimprove/proposers.py	89.56%	163 / 182
pyhealth/autoimprove/report.py	93.47%	558 / 597
pyhealth/autoimprove/space.py	97.40%	187 / 192
pyhealth/metrics/__init__.py	100.00%	12 / 12
pyhealth/metrics/clinical.py	100.00%	94 / 94
pyhealth/tasks/__init__.py	88.57%	31 / 35
pyhealth/tasks/base_task.py	83.33%	30 / 36
pyhealth/tasks/task_card.py	98.04%	50 / 51

Workflow run · a3752fb