Repository navigation
Conversation
Tika silently truncated extracted text at 500,000 characters, changing Text-Code and granular features for longer documents. Set a fixed 10M character cap - a constant rather than a configurable option so that independent parties derive identical ISCCs - and log a warning when the cap is reached. Also ignore the .agents directory.
Refresh uv.lock with current releases (xxhash 4.0.0, pypdf 6.16.1, pypdfium2 5.13.0, resvg-py 0.4.0, numpy 2.5.2, bitarray 3.10.1 and transitive/dev updates) and document the 0.9.6 changes in the changelog.
iscc-lib 0.7.0 fixes Image-Codes longer than 64 bits to follow ISO 24138:2024 (matching iscc-core 1.5.0). 64-bit Image-Codes and ISCC-CODEs are unaffected. iscc-core is not pinned because the SDK no longer depends on it. Refresh uv.lock (Pillow 13.0.0, exiv2 0.19.2, pypdf 6.21.1, and others) and update test expectations: - 256-bit Image-Code unit changes beyond the first 64 bits (iscc-lib) - image_meta_embed output differs at the byte level (exiv2 0.19.2) - EPUB cover thumbnail differs slightly (Pillow 13)
iscc-sci annotates its return value as dict[str, str], so the type checker matched the unpacked keys against model_construct's _fields_set parameter and failed the pre-push hook whenever the sci extra was installed. Passing the default explicitly resolves it without changing behavior.
deepwiki.com/badge.svg now sits behind a Vercel bot checkpoint that answers non-browser requests with an HTML challenge page (HTTP 429), so GitHub's image proxy and PyPI render it as a broken image. The badge still links to the DeepWiki page.
Parse container.xml and OPF files from EPUB archives through parse_xml_safe(), which disables entity resolution and network access, as svg.py already does. Defense in depth: lxml already refuses external entities by default. Fixes #156
Windows XP* tags such as XPTitle hold UCS-2 bytes that toString()
renders as a byte listing ("77 0 121 0 ..."). Read them via print(),
which decodes them to text.
Fixes #163
Struct containers stringify as the placeholder 'type="Struct"', which ended up as name or description and in the Meta-Code. Skip them so the next mapped field or the file name applies. Fixes #165
exiv2 returns bytes that are not valid UTF-8 as lone surrogates, which made code_meta fail with UnicodeEncodeError. Re-decode such values as Latin-1, the IPTC default without a UTF-8 CodedCharacterSet; values that cannot be repaired are skipped with a warning. Fixes #164
libmagic sniffs every MP4-family file as video/mp4, so .m4a files were processed as video and failed with "No video stream detected". Add audio/mp4 as a supported audio type (normalizing audio/x-m4a and registering .m4a), so the file name routes M4A to audio mode. TagLib silently drops MP4 tag names that contain a colon, so metadata is embedded in M4A files as ISCC_* tags, which extraction also reads. Sound-only .mp4/.mov files are not covered yet. Refs #166
The final CDC chunk can be arbitrarily small, and its simprint carries too little evidence to be useful. Drop a final chunk shorter than the CDC minimum chunk size (avg // 4) unless it is the only chunk. Retained simprints, offsets and sizes are unchanged. Fixes #162
text_features read the global text_avg_chunk_size instead of its per-call options, and code_text did not forward its options to text_features, so per-call text_avg_chunk_size and byte_offsets had no effect on granular features.
Bump version 0.9.5 -> 0.9.6 (pyproject, uv.lock, version test) and date the changelog.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #167 +/- ##
=======================================
Coverage 99.73% 99.73%
=======================================
Files 23 23
Lines 1890 1911 +21
=======================================
+ Hits 1885 1906 +21
Misses 5 5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Release v0.9.6. See the changelog section in this branch for full details.
iscc-lib>=0.7.0: Image-Codes longer than 64 bits now follow ISO 24138:2024XP*tags (EXIF XPTitle extracted as raw byte list instead of text #163), XMP structs (XMP struct extracted astype="Struct"instead of being skipped #165), non-UTF-8 IPTC crash (code_meta crashes on IPTC values that are not UTF-8 #164)code_iscctreats M4A and sound-only MP4 files as video and raises "No video stream detected" #166)text_avg_chunk_size/byte_offsetsfor granular text featurescode_image_semantic, DeepWiki badge via shields.ioCI passes on Linux, macOS and Windows for Python 3.11 to 3.14.
Fixes #156, fixes #162, fixes #163, fixes #164, fixes #165