Skip to content

feat: hvantk training-set build — labelled, tiered ML training datasets - #484

Merged
enriquea merged 15 commits into
devfrom
feature/training-set-build
Nov 12, 2026
Merged

enriquea merged 15 commits into
devfrom
feature/training-set-build

Conversation

@enriquea

Copy link
Copy Markdown
Member

Summary

Adds hvantk training-set build, which builds a labelled ML training dataset from one feature spec. Each feature column gets a leakage tier, L1 to L4. The command replaces the library-only hvantk/tools/training_sets/ path, which read pre-plugin tables by path and added hand-weighted composite scores.

Rows and labels. The rows are ClinVar GRCh38 missense SNVs:

  • labelled 1 (P, LP, P/LP) or 0 (B, LB, B/LB) from the primary CLNSIG (the text before the first |);
  • with review stars from CLNREVSTAT, re-joined at commas because import_vcf splits it;
  • filtered to at least one review star.

Columns. The spec declares them:

  • variant: allele-keyed scores and frequencies (dbNSFP rankscores and gnomAD AF), plus interval and codon flags (INSIDER interfaces, UniProt PTM sites);
  • gene: from an annotate compose matrix built from the same spec;
  • gene lists: membership flags, e.g. is_chd_gene.

Gene choice. Each row gets one gene: ClinVar GENEINFO Entrez id → HGNC → Ensembl. When ClinVar lists several genes, the first one dbNSFP also assigns to the variant wins, else the first listed.

Tiers are the higher of two:

  • Derived from the plugin's scores: trained_on: L2 if trained on the label's sources (ClinVar, HGMD, OMIM, ...), L3 if trained on population_af, otherwise L1.
  • Declared in the spec with tier:: L4 for an allele frequency or a score that takes one as input, L3 for constraint, L2 for e.g. simulated-allele predictors.

A declaration can raise a tier but never lower it, so it cannot hide circularity. That includes columns built from several scores and gene axes that aggregate scores. A column with neither a declaration nor a trained_on is left out and listed in the manifest.

Outputs: <prefix>.ht, a deterministic <prefix>.tsv.gz, <prefix>.dictionary.tsv (one row per column with tier, level, source, version, non-null and true rates) and <prefix>.manifest.json (funnel counts, label balance, gene choice, tiers, exclusions, gene-list mapping, input provenance).

Changes

  • New: hvantk/tools/training_sets/{build,training_set_cli}.py and training_set_cli.tool.yaml. The NAME=PATH option parser is shared with annotate compose through hvantk/tools/_cli_utils.py.
  • New: algorithms/annotation/tiers.py, a pure-Python tier rule built on rerank/provenance.resolve_arms.
  • New: algorithms/annotation/variant.py, with field selection, PTM codon loci and the multi-gene choice.
  • Feature spec: optional label, variant, gene and gene_lists sections, plus a tier per layer1 axis. Existing specs are unchanged.
  • annotate prepare: a matrix source may be keyed by Ensembl gene_id (version suffixes are dropped and no --hgnc is needed).
  • Plugins: DatasetManifest.score_provenance exposes a plugin's trained_on block, and PluginRegistry.get_manifest is new.
  • ClinVar: label_clinvar_variants replaces apply_clinvar_training_labels.
  • Fix in .tsv.gz export: the export of a hail table read Hail's temporary file back line by line through hl.hadoop_open. That ran at about 0.2 MB/s and stalled for good on a 210,867 x 312 table. The file is now read from local disk, and copied there once when the temp directory is not local. The output is byte-for-byte unchanged; a 20,000 x 307 table takes 48 s instead of 552 s.
  • Example: examples/training_sets/clinvar_heart.yaml, a ClinVar x heart dataset, with a README of the build commands.
  • Docs: docs_site/tools/training-set.md, a README row and the CHANGELOG.

Breaking changes

  • ClinVarVariantTableStreamer.label_training_set() now returns a 1/0 label with clinvar_review_stars, for missense SNVs. The rf_label TP/TN column and the gene_set, disease_terms and label_column options are removed. ptm constraint still reads a TP/TN column, and its page shows how to derive one.
  • Removed:
    • TrainingSetBuilder and build_clinvar_training_set;
    • algorithms/annotation/annotator.py, with its composite scores;
    • core/io/legacy_artifacts.py;
    • examples/training_sets/generate_enhanced_training_set.py.

Verification

  • Real data. I ran the full pipeline on the example spec with real inputs: ClinVar 2026-11-08, dbNSFP 4.9a, gnomAD constraint v2.1.1, GeVIR, INSIDER, UniProt PTM, HGNC, Ensembl 113, and three heart expression atlases.
    • Determinism: two complete builds wrote byte-identical .tsv.gz files.
    • Against an independent prototype (written directly in Hail and pandas): the same 210,867 variants, and every column equal per variant except gene_id/gene_symbol on 3,020 multi-gene rows, where the new gene rule applies by design.
    • Gene columns: exactly equal, at float64, per gene for all 14,963 genes.
  • Funnel: 4,617,349 ClinVar records → 2,595,328 missense SNVs → 227,513 labelled → 210,867 with at least one star.
  • Labels: 62,479 P/LP vs 148,388 B/LB.
  • Columns: 312 in total, 301 of them features. Tiers are L1 265, L2 24, L3 8, L4 4, and nothing is excluded.
  • Build time: about 5 minutes for the build and the TSV together.
  • Tests: the fast suite and the Hail-marked annotation, ClinVar and TSV tests pass. Four pqtl caplog tests fail in a full run, but they fail the same way on dev (order-dependent) and pass alone.
  • One new regression test: CLNREVSTAT must be re-joined before stars are read.

enriquea and others added 13 commits November 11, 2026 18:34
DatasetManifest.score_provenance carries the plugin's scores: block
(score field -> trained_on sources); PluginRegistry.get_manifest returns
one dataset's descriptive manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An atlas whose genes are Ensembl ids (e.g. a bulk Expression Atlas
experiment) can now go through annotate prepare as a matrix entry with
key: gene_id; version suffixes are dropped and no --hgnc is needed.
key: symbol behaves as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Columns get L1-L4 from their declared trained_on (via resolve_arms
against the label's sources) and explicit spec declarations; a column
with neither is excluded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
label_clinvar_variants replaces apply_clinvar_training_labels: missense
SNVs labelled from the primary CLNSIG (text before the first '|'),
review stars from the comma-re-joined CLNREVSTAT, GENEINFO genes kept in
order. label_training_set now returns that 1/0 label; its TP/TN column
and gene_set/disease_terms gating are removed with the legacy
training-set builder they served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The feature spec gains label, variant, gene and gene_lists sections and
a tier per layer1 axis, all optional, so annotate specs are unchanged.
Cross-section checks run when the spec loads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dotted field selection with float casts and max-reduction for
allele-keyed sources, PTM codon loci, and the multi-gene choice
(first candidate a second source also assigns, else the first).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_training_set labels ClinVar rows, joins the spec's variant sources
(allele columns, interval and codon flags), chooses each row's gene,
joins the composed gene matrix and gene-list flags, resolves tiers and
measures every column. write_training_set writes the .ht and the
deterministic .tsv.gz with provenance, the feature dictionary and the
manifest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The command reads the spec, checks every --input and --gene-list name
against it before Hail starts, takes each axis's columns from the
compose manifest and each variant source's trained_on from its plugin
manifest, then builds and writes the training set. The NAME=PATH option
parser is shared with annotate compose. The TSV is written through
core.io.save (save_native has no .tsv.gz path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
examples/training_sets/ now holds the ClinVar x heart spec and the
commands that build it. The annotators, their composite scores and the
legacy artifact loader go; docs gain the training-set page, the README
row and the changelog entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hl.hadoop_open crosses into the JVM for every read: the export came
back at about 0.2 MB/s and, on a 210,000 x 312 table, stalled for good
after an hour. The export is now opened directly when Hail's temporary
directory is local, or copied to a local temporary directory once.
Output is byte-for-byte unchanged (checked against the previous reader);
20,000 x 307 takes 48 s instead of 552 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A column built from several scores (fields + reduce) and the score
  columns of a layer1 axis that aggregates per-variant scores now take
  their inputs' trained_on (union; undeclared if any input is), so a
  declared tier can no longer hide circularity there; the CLI fetches
  provenance for aggregated axes too.
- The manifest lists every unmapped gene-list symbol (the mapper's
  report names at most 20) with counts that reconcile: symbols in,
  symbols mapped, genes, unmapped.
- The TSV read-back treats a temp path with no scheme as local only if
  it exists on this disk; otherwise (Hadoop default filesystem, e.g.
  HDFS) it is copied once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A tier is now the higher of the one derived from trained_on and the
  spec's declaration, so a declaration can raise a tier but never lower
  it; a column trained on ClinVar and population_af is L3. The heart
  spec's 301 feature tiers are unchanged.
- The CLI runs the input checks (now also codon fields and the HGNC
  fields the builder reads) before loading anything and reports a
  mistake as a usage error, as the docs say.
- The dictionary's label description lists the classes with ', '.
- The package docstring no longer mentions the removed annotators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown
Contributor

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c30d9aeb-77a6-47dd-8658-15026bd0121e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codacy-production

codacy-production Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 11 medium · 28 minor

Alerts:
⚠ 39 issues (≤ 0 issues of at least minor severity)

Results:
39 new issues

Category Results
Documentation 27 minor
CodeStyle 1 minor
Complexity 11 medium

View in Codacy

🟢 Metrics 251 complexity

Metric Results
Complexity 251

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

- resolve_tiers takes equivalence=None instead of a dict default.
- build_training_set documents spec_sha256 and legacy_names.
- The example README no longer has a literal tab inside inline code.
@enriquea

Copy link
Copy Markdown
Member Author

Codacy triage (51477d9):

  • Fixed: the mutable dict default on resolve_tiers (equivalence=None now); the missing spec_sha256/legacy_names parameter docs on build_training_set; a literal tab inside inline code in examples/training_sets/README.md.
  • Not changed:
    • Complexity (11): the repo's own limit (ruff mccabe, max 10) passes on every new module. reduce_matrix_to_gene predates this PR.
    • Docstring layout (D213, D413; 26): these ask for the opposite of the layout the rest of the codebase uses.
    • D301: \\ in the CLI module's example is a valid escape, and no module warns on import.
    • "Lambda may not be necessary": kept for readability.

@enriquea
enriquea merged commit f51ec6c into dev Nov 12, 2026
9 of 10 checks passed
@enriquea
enriquea deleted the feature/training-set-build branch November 12, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant