[solarsys]

Stacked on #1318, then #1317 and #1316. Review the last commit, autoimprove: features as data.

Why

Model and setting search reaches a ceiling quickly; better inputs are where the remaining gains are. Changing inputs must not mean running code that an LLM (or anyone) wrote, on a machine that holds patient data. A feature also has to be reviewable: a clinician must be able to read exactly what was computed, and know that it cannot see the future.

What changes

• FeatureSpec / FeatureSpace. A feature is JSON built from a fixed set of operators over PyHealth's event log (Patient.get_events), evaluated at each sample's prediction time t:

  • over a window that ends strictly before t: count, exists, days_since, stat (min, max, mean, last or slope of a value), n_distinct, trend and blocks_with;
  • combinations: ratio, diff, band, ge, any and cross;
  • column, for an existing task feature.

  Selectors name an event type, an optional vocabulary group (e.g. SABA) and optional where filters. Filter values must occur for at least 11 patients.

• Definitions are generated from the spec, deterministically. For example, ge(3, count(365 d, SABA orders)) becomes "1 if there were 3 or more SABA orders in the 365 days before the landmark, else 0". So the sentence a clinician reviews is exactly what was computed. Every spec must also carry its clinical rationale and expected direction.

• Refusals carry a reason, the same way invalid configs already do:

  • an unknown operator, group or attribute;
  • a window outside the menu;
  • nesting deeper than 3;
  • a missing rationale;
  • more than 8 features per proposal.

• The future-blind test. Each spec is computed twice: on the full record, and on the record as known at t. Rows dated at or after t are removed, and values known only later (known_at, e.g. a stay's length known at discharge) are blanked. Derived events are rebuilt from the truncated record. Any difference rejects the spec.

• check() reports each spec's definition, coverage and future-blind result. augment() adds an engineered tensor field (EngineeredDataset). run_config_cv now imputes and scales that field on each fold's training part (fold_parts), so a held-out fold never informs its own medians and scales. The other fields keep the processors set_task fitted, as before.

• Optional BaseTask hooks: time_key, feature_vocabulary(), and derived_events(patient, t) with derived_event_types. They are class-level defaults, so set_task cache keys (built from vars(task)) are unchanged, and tasks without them work as before.

• Speed: each compute call fetches the needed event types for all its patients in one query.

Example and checks

examples/autoimprove_features_mimic3.py checks one hand-written feature on the full synthetic MIMIC-III, with no search and no LLM. The feature is hospital admissions in the 365 days before discharge, used to predict next-visit mortality:

count_365d__admissions: the number of hospital admissions in the 365 days before discharge
  rationale: frequent recent hospital stays mark frailer patients; expected direction +
  coverage 76%; future-blind: passed
CV ROC-AUC without the feature 0.532, with it 0.557: gain +0.025 ± 0.020

(It runs in about 20 seconds. Coverage is below 100% because a third of the synthetic stays end before they begin.)

Tests

tests/core/test_autoimprove_features.py has 10 tests:

• operators: every one checked against hand-computed values on a small event log, including window boundaries, missing values and the slope;
• definitions: matched exactly;
• refusals: each with its reason;
• round trip: JSON → validate → compile → name and definition;
• planted leaks: both rejected by the future-blind test (a length of stay known only after t, and a derived event built from all events), while their correct versions pass;
• per-fold statistics: fitted on the training part only.

The earlier autoimprove and metrics tests still pass (48 in total), as do the doctests, ruff and tools/check_pr_rules.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ntMLEmFsQEQJKTmPGGw4T

[github-actions]

Test Coverage

Total: 70.48% (19280 / 27357 lines)

Modified file	Coverage	Lines
pyhealth/autoimprove/__init__.py	100.00%	8 / 8
pyhealth/autoimprove/evaluate.py	92.49%	160 / 173
pyhealth/autoimprove/explain.py	86.63%	175 / 202
pyhealth/autoimprove/features.py	90.91%	510 / 561
pyhealth/autoimprove/improver.py	93.28%	541 / 580
pyhealth/autoimprove/proposers.py	93.91%	108 / 115
pyhealth/autoimprove/report.py	93.23%	468 / 502
pyhealth/autoimprove/space.py	97.40%	187 / 192
pyhealth/metrics/__init__.py	100.00%	12 / 12
pyhealth/metrics/clinical.py	100.00%	94 / 94
pyhealth/tasks/__init__.py	88.57%	31 / 35
pyhealth/tasks/base_task.py	83.33%	30 / 36
pyhealth/tasks/task_card.py	98.04%	50 / 51

Workflow run · e025a02