Skip to content

fix: read the hail TSV export back as UTF-8, not ISO-8859-1 - #483

Merged
enriquea merged 1 commit into
devfrom
fix/tsv-utf8-read
Nov 11, 2026
Merged

enriquea merged 1 commit into
devfrom
fix/tsv-utf8-read

Conversation

@enriquea

Copy link
Copy Markdown
Member

The TSV export from #482 (8683258) writes every non-ASCII character double-encoded when the table is a hail table: RORα becomes RORα.

Cause

_hail_rows reads Hail's temporary export with hl.hadoop_open(tmp, "r"). Hail writes that file as UTF-8, but its text mode wraps the handle in io.TextIOWrapper(..., encoding='iso-8859-1') (hail/fs/hadoop_fs.py, hail 0.2.137). Each UTF-8 byte is read as its own character and written to the .tsv.gz re-encoded. The pandas backend was not affected.

Fix

Open the file in binary and decode it as UTF-8 (hvantk/core/io/_tsv.py). Decoding is strict, so a byte that is not UTF-8 raises instead of being misread.

Measured

  • Test: the hail test in test_annotation_table_tsv.py now carries RORα in an array element. With the old read it fails on 50%25|RORα. With the fix, all 15 tests in the file pass, hail included. ruff check and ruff format --check are clean.
  • Real data: reprocess hgnc:lookup over the 2026-10-06 HGNC archive.
    • At 8683258, 35 alias_symbols cells were corrupted, exactly the export's non-ASCII cells (e.g. HGNC:10258).
    • At this commit, the .tsv.gz is byte-identical to the export from 4c90a24, before the writer was rewritten (sha256 67bbbc5a…).
    • The feeds HGNC recipe cross-checks every cell against HGNC's raw file. It refused the 8683258 export and accepts this one with 0 mismatches.
  • The one other text-mode read (skills/clingen/builder.py:75) also writes in text mode, so it round-trips bytes unchanged under ISO-8859-1. Left as is.

No CHANGELOG entry, because the TSV export is still under Unreleased and its entry describes the fixed behaviour.

The commit that merges this becomes built_by.hvantk_release in the feeds HGNC package.

Hail writes the temporary export as UTF-8, but hl.hadoop_open(path, "r")
wraps it in a TextIOWrapper decoding ISO-8859-1 (hail/fs/hadoop_fs.py,
hail 0.2.137), so every non-ASCII character was written to the .tsv.gz
double-encoded: "RORα" became "RORα". Measured on the 2026-10-06 HGNC
archive through hgnc:lookup: 35 alias_symbols cells corrupted, exactly the
export's non-ASCII cells; the pandas backend was never affected.

The file is now opened in binary and decoded as UTF-8 (strict, so a byte
that is not UTF-8 raises instead of being misread).

The hail test now carries "RORα" in an array element. With the old read
it fails on "50%25|RORα"; with the fix all 15 tests in
test_annotation_table_tsv.py pass (hail included).
@qodo-code-review

Copy link
Copy Markdown
Contributor

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2b119118-373f-4dab-acc1-2ea15372a504

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 complexity

Metric Results
Complexity 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@enriquea

Copy link
Copy Markdown
Member Author

The three red checks never reached a test. Plugin contract (hail), HGC pipeline (hail) and build-linux all fail at their first step, sudo apt-get install -y libopenblas-dev, with exit 100: "Temporary failure resolving 'security.ubuntu.com'". This happened on the first run and again on a re-run of the failed jobs, and the same three jobs passed on dev at 8683258.

Their test commands, run locally at this PR's head (ff047b9; hail 0.2.137, Python 3.12):

  • Plugin contract (hail): the workflow's exact pytest … -m "hail or round_trip_contract" --require-round-trips gives 118 passed.
  • HGC pipeline (hail): pytest hvantk/tests/hgc -m hail gives 22 passed.
  • build-linux: this runs the default pytest --require-round-trips, which passed in build (3.10), (3.11) and (3.12) on this PR.

@enriquea
enriquea merged commit a7ce830 into dev Nov 11, 2026
9 of 15 checks passed
@enriquea
enriquea deleted the fix/tsv-utf8-read branch November 11, 2026 17:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant