[solarsys]

Stacked on #1317, which is stacked on #1316. Until those merge, the diff includes them; review the last commit, autoimprove: clinical explanation, clinical performance and review.

Why

#1317 explains why a model was selected. A clinician also needs to know three more things:

• what the model relies on;
• whether what it learned is plausible;
• how it would perform if acted on.

This PR adds those, plus the items a clinician must sign off before use.

What changes

• pyhealth.tasks.TaskCard and an optional BaseTask.task_card() hook (default None). The card records:

  • the cohort, prediction time, outcome, horizon and exclusions, each with its rationale;
  • a feature dictionary: meaning, clinical group and expected direction of effect (+, -, U-shaped, or none).

• pyhealth.metrics.clinical, usable on its own, for example by the clinical benchmark's metrics step (I-03):

  • alert_rate_metrics: sensitivity, PPV and number needed to evaluate when the top 5, 10 or 20% of risk is flagged;
  • net_benefit: the decision curve;
  • calibration_bins and subgroup_metrics;
  • clinical_summary, which runs all of the above.

  Any count below 11 patients is suppressed (the HIPAA/CMS small-cell convention), counted in distinct patients.

• AutoImprover(task_card=...) and explain(). Attributions come from exact TreeSHAP for trees and from integrated gradients for torch models, which is exact for logistic regression; numeric tensor and time-series fields are covered. explain() reports:

  • importance by clinical group;
  • the top features with their meaning and effect shape, in bins of at least 11 patients whose cut points are midpoints, not patient values;
  • learned against expected direction;
  • how many top features the fold models share.

• evaluate_on_test(subgroups=...) adds the clinical summary for each config.

• flags() / approve(). Flags cover:

  • a missing card;
  • a top feature the card doesn't describe;
  • a learned effect opposite to the expected one;
  • poor calibration;
  • a subgroup gap.

  Decisions are recorded in review.json.

• The report gains How it decides, Performance in clinical terms, Clinician review, and a checklist of where it covers the main TRIPOD+AI items.

Tried on the eICU 48 h mortality task (open demo data)

The example now has a task card for all 88 tabular columns, plus age-band and sex subgroups.

• Claude's selected logistic regression: test PR-AUC 0.328. Flagging the top 10% of risk catches 48% of deaths with 34% PPV, against 30% for the XGBoost baseline.
• Calibration flag: this model uses class weighting, so its risks are inflated (observed/expected 0.42). The flag fires, and the net-benefit table shows the cost: below the baseline at action thresholds until recalibrated.
• Direction flags on monitoring counts: the number of SpO₂, respiratory-rate and heart-rate measurements was learned as protective, opposite to the card. In eICU these vital signs are charted automatically every 5 minutes, so the count measures time in ICU before 48 h, not severity. The flag caught a wrong expectation in the task card itself. That is the clinician review step working as intended.
• Time-series view: RETAIN's explanation is per lab and vital variable. Age, lactate, urea nitrogen and low blood pressure all came out in the expected directions.

Tests

• New tests:
  • clinical metrics: against hand-computed values, small-cell suppression, and calibration bins merging to 11 patients;
  • task card: a round trip;
  • planted reversed feature: it is flagged, approved, and the approval shows in the report;
  • attribution: logistic-regression attributions equal coefficient × (value − mean) to within 1e-4, and TreeSHAP gives the same directions;
  • privacy: the report contains no patient IDs and no cell under 11.
• 38 autoimprove and metrics tests pass, as do the doctests, ruff and tools/check_pr_rules.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ntMLEmFsQEQJKTmPGGw4T

[github-actions]

Test Coverage

Total: 70.04% (18755 / 26778 lines)

Modified file	Coverage	Lines
pyhealth/autoimprove/__init__.py	100.00%	7 / 7
pyhealth/autoimprove/explain.py	86.63%	175 / 202
pyhealth/autoimprove/improver.py	93.28%	541 / 580
pyhealth/autoimprove/proposers.py	93.91%	108 / 115
pyhealth/autoimprove/report.py	93.23%	468 / 502
pyhealth/autoimprove/space.py	97.40%	187 / 192
pyhealth/metrics/__init__.py	100.00%	12 / 12
pyhealth/metrics/clinical.py	100.00%	94 / 94
pyhealth/tasks/__init__.py	88.57%	31 / 35
pyhealth/tasks/base_task.py	85.71%	24 / 28
pyhealth/tasks/task_card.py	98.04%	50 / 51

Workflow run · d1e5f76