Repository navigation
fix: read the hail TSV export back as UTF-8, not ISO-8859-1 - #483
Conversation
Hail writes the temporary export as UTF-8, but hl.hadoop_open(path, "r") wraps it in a TextIOWrapper decoding ISO-8859-1 (hail/fs/hadoop_fs.py, hail 0.2.137), so every non-ASCII character was written to the .tsv.gz double-encoded: "RORα" became "RORα". Measured on the 2026-10-06 HGNC archive through hgnc:lookup: 35 alias_symbols cells corrupted, exactly the export's non-ASCII cells; the pandas backend was never affected. The file is now opened in binary and decoded as UTF-8 (strict, so a byte that is not UTF-8 raises instead of being misread). The hail test now carries "RORα" in an array element. With the old read it fails on "50%25|RORα"; with the fix all 15 tests in test_annotation_table_tsv.py pass (hail included).
|
ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
The three red checks never reached a test. Their test commands, run locally at this PR's head (ff047b9; hail 0.2.137, Python 3.12):
|
The TSV export from #482 (8683258) writes every non-ASCII character double-encoded when the table is a hail table:
RORαbecomesRORα.Cause
_hail_rowsreads Hail's temporary export withhl.hadoop_open(tmp, "r"). Hail writes that file as UTF-8, but its text mode wraps the handle inio.TextIOWrapper(..., encoding='iso-8859-1')(hail/fs/hadoop_fs.py, hail 0.2.137). Each UTF-8 byte is read as its own character and written to the.tsv.gzre-encoded. The pandas backend was not affected.Fix
Open the file in binary and decode it as UTF-8 (
hvantk/core/io/_tsv.py). Decoding is strict, so a byte that is not UTF-8 raises instead of being misread.Measured
test_annotation_table_tsv.pynow carriesRORαin an array element. With the old read it fails on50%25|RORα. With the fix, all 15 tests in the file pass, hail included.ruff checkandruff format --checkare clean.reprocess hgnc:lookupover the 2026-10-06 HGNC archive.alias_symbolscells were corrupted, exactly the export's non-ASCII cells (e.g. HGNC:10258)..tsv.gzis byte-identical to the export from 4c90a24, before the writer was rewritten (sha25667bbbc5a…).skills/clingen/builder.py:75) also writes in text mode, so it round-trips bytes unchanged under ISO-8859-1. Left as is.No CHANGELOG entry, because the TSV export is still under Unreleased and its entry describes the fixed behaviour.
The commit that merges this becomes
built_by.hvantk_releasein the feeds HGNC package.