Skip to content

feat(skills): ml-features — build a labelled, leakage-tiered ML training set on a cluster - #485

Merged
enriquea merged 20 commits into
devfrom
feature/ml-features-skill
Nov 12, 2026
Merged

enriquea merged 20 commits into
devfrom
feature/ml-features-skill

Conversation

@enriquea

Copy link
Copy Markdown
Member

Summary

Adds the ml-features workflow skill. It tells a coding agent how to build a labelled, leakage-tiered ML training set with hvantk training-set build (#484) on a Slurm cluster:

  • stage and build the inputs, with the ClinVar release pinned;
  • compose the gene matrix from the same spec;
  • build the training set;
  • write a receipt that checks the four outputs agree and records the TSV and sidecar digests.

Its references say how to adapt the spec, which tier to declare, and how to pick training columns free of label or allele-frequency leakage.

Running the skill's eval control found a toolkit bug, fixed here: hvantk download clinvar --version YYYYMMDD failed for every date.

Changes

  • New skill: skills/ml-features/, following skills/workflow-conventions/SKILL.md:
    • SKILL.md, with the six sections; stage: analysis, scale: node;
    • references/inputs.md: where each input of the example spec comes from and how it is built, including the two with no downloader (dbNSFP and the INSIDER variants BED). It also covers the dbNSFP rankscore-type check, making dbNSFP's single .bgz without bgzip, and the UniProt PTM build's Ensembl GTF on clusters whose compute nodes have no internet;
    • references/spec-and-tiers.md: adapting the spec, declaring tiers, and the four training arms with the selection code;
    • templates/describe-training-set.py: the receipt. It exits 1 when the .ht, the TSV, the dictionary and the manifest disagree, or when a feature column is typed str (a dbNSFP built by an older checkout passes text through without any error).
  • Eval: skills/ml-features/eval/ is a chromosome-22 slice of the ClinVar x heart example without its three expression axes:
    • stage.sh stages public inputs plus a dbNSFP file the user supplies, which is academic-licensed with no download URL. UniProt PTM is a live snapshot with no archived releases, so the reference records a digest of the staged inputs and the control asks for a re-baseline when they differ;
    • control.sh runs a positive and a negative control. The negative uses the staged spec with min_review_stars: 0. raw/ and upstream/ are left read-only, so a run under test cannot change the stage;
    • check.py grades M1-M5 through the skill's own receipt template;
    • reference.json was written by the control on a cluster.
  • Fix: dated ClinVar downloads. The downloader looked in vcf_<build>/archive/, a directory NCBI doesn't have, so every dated download returned 404. A dated release now resolves to archive_2.0/<year>/, weekly/, or the top level while it is current. The candidates are probed with HEAD requests, and the .md5 and .tbi come from the same place, also when verify_md5() is called on its own. A timeout while probing is reported with the URL. The docs' example date was not a release date.
  • Fix: training-set provenance. A training set's sidecar always had builder_commit empty. It now records the commit that built it, through the same resolve_builder_commit() every dataset build uses.
  • Docs:
    • the skills/README.md catalog row and the README skill count and badge (12 → 13);
    • the workflow-skills guide and the plugin descriptions;
    • a "See also" link on the training-set page;
    • the CHANGELOG.
  • Example README: the build steps now give the real prepare command for each axis. The old loop passed one placeholder input to every axis. The steps also add the two missing expression summaries and say to pin ClinVar.
  • Gene-index skill: its references/annotation-axes.md and one trap were stale since feat: hvantk training-set build — labelled, tiered ML training datasets #484. They now cover the training-set sections, matrix: entries keyed by gene_id (version suffixes dropped), and tier.

Verification

  • Skill tests: test_workflow_skill_conformance.py and test_no_site_data_in_skills.py pass, including the catalog/badge check.

  • Eval on a cluster: the control ran four times: once with WRITE_REFERENCE=1, then three times without it, the last two after the review fixes.

    • Both arms passed every run: the positive passes M1-M5, and the negative fails on content only (M4 true, content_ok false, reported_ok true).
    • Every run produced the same TSV digest.
    • Reference: 99,994 chr22 ClinVar records give 4,157 labelled rows with at least one star (870 positive, 3,287 negative), with tiers L1 26, L2 24, L3 8, L4 4.
    • Every job fits the eval budget of 4 cores, 16 GB and 45 minutes. The longest took 3 min 14 s, the peak RSS was 3.2 GB, and the positive arm took about 12 minutes end to end.
  • Receipt template:

  • ClinVar downloads:

    • --version 20261108 (the current release) downloads from weekly/, and the MD5 matches;
    • 20261004 resolves to archive_2.0/2026/;
    • a date with no release fails with the three URLs tried.
    • The downloader tests are updated for the new layout. The CLI test now runs the real probe, with a 404 in archive_2.0/ falling through to weekly/ for both the VCF and its index.
  • Skill tested with agents: two fresh agents were given realistic tasks without the skill, then again with it. Without it, both downloaded ClinVar latest, guessed job sizes 4-25x too large, and wrote ad-hoc verification scripts. With the skill, both pinned ClinVar, used the measured sizes and the receipt template, and B gave the four training arms with gene-grouped folds. They also found four gaps in references/inputs.md, all checked and fixed here:

    • the input grep missed - input: list items;
    • the PTM build downloads the Ensembl GTF on the compute node;
    • hvantk download ucsc --search needs --list-datasets;
    • neither the image nor the host has bgzip for dbNSFP.

    After the whole-branch review's fixes, a third fresh agent re-ran scenario A. It passed every checklist item, and its refinements were checked against the code and fixed:

    • --hgnc for the symbol-keyed axes;
    • the dbNSFP job stops on an empty glob and writes the file atomically;
    • ClinVar is staged with --verify-md5;
    • the UCSC download cancels on a non-empty directory;
    • the training-set sidecar had no commit.
  • Whole-branch review by a fresh reviewer: no critical findings. Its important and minor findings are fixed here, apart from two toolkit checks left to the procedure (the build does not compare a matrix axis's columns with the spec, and the ClinVar dataset does not record its release; the skill says both).

  • Test suite (CI's selection, pytest --require-round-trips): 3,592 passed. Four pqtl log-capture tests fail in a full local run; they pass when run alone and in CI, and fail the same way on dev.

SKILL.md with the six-section contract, references for the inputs and for
the spec and tiers, and a receipt template that checks a training set's
four outputs agree and binds them to the TSV and sidecar digests.
A chromosome-22 slice of the ClinVar x heart example without its expression
axes, from public inputs plus a dbNSFP file the user brings: stage.sh,
a positive and a negative control, check.py (M1-M5, grading through the
skill's receipt template) and a placeholder reference.json until the
control writes it.
hvantk download clinvar --version YYYYMMDD built vcf_<build>/archive/<file>,
a directory NCBI does not have, so every dated download returned 404. A
dated release is in archive_2.0/<year>/ once archived, in weekly/, and at
the top level while current: probe them in that order with HEAD requests
and download the first that exists (its .md5 and .tbi from the same place).
The docs' example date was not a release date.
The README's skill count and badge (13), the workflow-skills guide, the plugin
descriptions and a see-also link on the training-set page. The training-set
example README gives the real prepare command per axis (the loop passed one
placeholder input to every axis), the two missing expression summaries and
the ClinVar pin. The gene-index skill's annotation-axes reference and one
trap are brought up to date with #484 (training-set sections, matrix entries
keyed by gene_id, tier).
…ldcard

nullglob leaves a literal file name in place when no such file exists, so the
INSIDER interfaces table counted as staged before it was downloaded.
A stage made on another day differs from the reference's, and the control's
input check then asks for a re-baseline; say why.
The UniProt PTM build downloads the pinned Ensembl GTF into its raw dir and, with
no --intermediate, writes its parse output there too. Both broke the next control
run's raw-dir check, and the GTF download fails where compute nodes have no
internet. stage.sh now copies the GTF from the gene-index stage, and the control
writes the intermediate under its own output.
- the spec input grep missed list items (- input:), finding 2 of 5 inputs;
- UniProt PTM: copy the pinned Ensembl GTF where compute nodes have no internet,
  keep the parse intermediate out of the raw dir, note the snapshot date;
- dbNSFP: make the one .bgz from the per-chromosome files without bgzip;
- Farah: the UCSC dataset is hoc/all-heart, listed with --list-datasets.
Written by the positive control on a cluster at c50bada; a second control run
without WRITE_REFERENCE passed both arms with the same TSV digest.
…clear error

_locate() keeps the URL it finds, so verify_md5() called on its own fetches the
.md5 from where the release is, not from the first candidate. A read timeout in
the HEAD probe raised a bare TimeoutError past the CLI's error handling; any
OSError now reports the URL it could not reach. The CLI test runs the real probe
(a 404 in archive_2.0/, then weekly/) instead of patching it out.
- the dataset does not record the ClinVar release (source_fingerprint is the
  upstream's state at build time): record the file, its sha256sum and ##fileDate
  in the run notes, and compare those;
- training-set build checks that each layer1 axis is in the matrix, not its
  columns: prepare and compose again after any layer1 edit; the wrong-groups
  error comes from annotate prepare, now a trap and in the description;
- traps for a GRCh37 ClinVar VCF and for CLNSIG/CLNREVSTAT commas;
- the receipt exits 1 on a feature column typed str;
- the dbNSFP .bgz is written by one pysam job, not a login-node cat;
- eval: the ClinVar file is taken by version, a failed curl stops staging, raw/
  and upstream/ are left read-only, the negative spec is generated from the
  staged one (spec_negative.yaml removed), stale comments fixed;
- plugin descriptions: a double 'and'.
…name

The login node's python3 can be too old to run hvantk/resources/ensembl_release.py.
builder_commit was always null on training-set sidecars, while every input dataset
records it through resolve_builder_commit(); use the same function.
- copy the spec and gene lists to $OUT first; stage ClinVar with --verify-md5 and
  keep the VCF; how to find a release date; read ##fileDate with zgrep (no
  SIGPIPE under pipefail);
- the dbNSFP .bgz job stops when no file matches and renames only when complete;
  the per-chromosome headers must be identical, with the check;
- --hgnc for every uniprot_id- or symbol-keyed axis (farah and hca too);
- expression summarize loads the whole .h5ad; Expression Atlas needs FTP; the
  UCSC download cancels on a non-empty directory;
- the GRCh38 trap is for position-keyed inputs; the provenance loop also finds
  .h5ad sidecars.
…sion

A job on a node without git records hvantk <version>; the commit is then only in
the run notes.
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a0658b00-4630-44ec-a556-4e50c6cfe295

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@qodo-code-review

Copy link
Copy Markdown
Contributor

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@codacy-production

codacy-production Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 critical · 8 medium · 27 minor

Alerts:
⚠ 36 issues (≤ 0 issues of at least minor severity)

Results:
36 new issues

Category Results
Documentation 25 minor
Security 2 minor
1 critical
3 medium
Complexity 5 medium

View in Codacy

🟢 Metrics 174 complexity

Metric Results
Complexity 174

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@enriquea

Copy link
Copy Markdown
Member Author

Codacy triage (202521c):

  • Fixed: eval/slice_text.py starts gzip by its resolved path (B607).
  • Not changed:
    • Complexity (5, Lizard): the repo's own limit (ruff mccabe, max 10) passes on every changed module. grade_slurm and main in eval/check.py keep the shape of the other skills' eval checks.
    • Docstring layout (D213, D209, D205, D415, D403; 25): these ask for the opposite of the layout the rest of the codebase uses, as on feat: hvantk training-set build — labelled, tiered ML training datasets #484.
    • subprocess (B404, B603; 4): fixed programs (gzip, sacct) with list arguments and no shell.
    • urlopen (B310; 1): the ClinVar HEAD probe builds its URLs from the fixed https NCBI base, like the existing .md5 fetch.

@enriquea
enriquea merged commit e23c052 into dev Nov 12, 2026
9 of 10 checks passed
@enriquea
enriquea deleted the feature/ml-features-skill branch November 12, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant