Repository navigation
Conversation
replace_text_in_document only accepted a single literal /FlateDecode filter, so content streams encoded with a chained filter array (e.g. ReportLab's [/ASCII85Decode /FlateDecode]) were skipped entirely (streams_scanned=0), leaving all entity text unredacted. Decode (apply_filter_chain_decode) and re-encode (apply_filter_chain_encode, preserving /Filter shape) already support these chains, so the gate is widened to any chain containing /FlateDecode; unsupported chains remain fail-safe (skip on decode, no-op on encode). Also guard collectPageFonts against get_keys() returning false on unparsable /Font dicts (was emitting foreach-on-false warnings).
…alueSimple Some producers emit /Resources or /Font as a bare "6 0 R" token that the parser stores as a PDFValueSimple rather than a PDFValueReference. The is_a(PDFValueReference) gates in collectPageFonts missed that form, so the font dict was never dereferenced (get_keys() === false) and the page's fonts were not collected. With no font info, resolveOperandToUnicode fell back to a raw-byte passthrough, leaving WinAnsi high bytes (e.g. 0xEB) unconverted — so accented needles (ë/ï/ü) never matched and that text was left unredacted. Add resolveIndirectValue() handling both reference forms (PDFValueSimple get_object_referenced() already parses "N G R") and use it for both the /Resources and /Font resolution. Fonts now resolve, accented text resolves to UTF-8 via the font's WinAnsi encoding, and matching + splicing work via the existing font-aware path.
parseContentStreamModel was the dominant cost of replace_text_in_document on large PDFs — measured at 89% of replace time (67.8s of 75.9s on a 50-page doc, 1 needle), because its inner loop called findNextOperator() once per operator TYPE (Tj/TJ/Tf/Tm) at every step. A type absent from the rest of a block scans forward to its next occurrence anywhere ahead, so the per-stream parse is O(n^2); it is also rebuilt 3x per stream (once per cross-phase). Add findNextAnyOperator(): the same string-state-aware tokenizer, but it returns the earliest operator of ANY requested type in a single forward scan, so each step advances only to the next operator (linear per block). Behaviour is identical — the old code already chose the earliest-position candidate. Result: replace_text on the 50-page fixture drops 76s -> 12.5s (1 needle) and a realistic 445-needle per-document run completes in ~14s (peak 14MB) where it previously exceeded 2 minutes.
… parse PDF comments (`%` to end-of-line) are whitespace-equivalent everywhere outside strings and stream data, so they may appear between dictionary entries. The value parser already skipped T_COMMENT, but the object/dict parser (_parse_obj) did not — a comment inside a dictionary hit `default: throw "Invalid token"`. ReportLab writes a `% ... -- digest (opensource)` line INSIDE the trailer dictionary, which made get_trailer() report "trailer is not valid", acquire_structure() fail, and PDFDoc::from_string() return false — so every ReportLab-generated PDF (very common) failed to load and could not be anonymised/text-replaced. Fix centrally in nexttoken(): transparently skip T_COMMENT tokens so every parse context (dictionaries, lists, values) tolerates comments. Strings, hex-strings and streams use dedicated readers and are unaffected. Verified: PDFDoc::from_string() now loads a ReportLab PDF that previously returned false.
buildFontInfo() flags a font's /Encoding as "unhandled" only when it is an
inline dictionary (/BaseEncoding + /Differences). The detection used
`$enc->get_keys() !== []`, but get_keys() returns `false` (not `[]`) for a NAME
value (PDFValueType). Since `false !== []` is true, a *named* encoding such as
`/Identity-H` was wrongly flagged as an unhandled dict, fell back to WinAnsi,
and never used the /ToUnicode CMap — so CID (Type0/CIDFontType2/Identity-H)
fonts matched nothing and replaced zero text.
This is exactly the Google Docs / Skia ("AAAAAA+ArialMT" subset, Identity-H)
case: text extracts fine (ToUnicode) but replace_text_in_document produced
streams_modified=0 and encoding_dict_unhandled for the font.
Guard the dict check with is_array() so only a real inline dict is treated as
unhandled; named encodings route to FontEncoding::forName() and use the
ToUnicode CMap. Verified on a Google-Docs PDF: "Rotterdam" ×9 -> 0 (9
placeholders, streams_modified=2), encoding_dict_unhandled cleared.
…) PDFs Skia / Google-Docs / PDF.js exports lay text out as one BT/ET block per word with one Tj per glyph, positioned via Td under a vertical-flip text matrix (d = -1). On these documents multi-word entities were never replaced (single words were), leaving PII residuals. Two compounding causes, both fixed: 1. groupBlocksIntoLogicalLines / groupBlocksIntoLinePairs rejected the blocks as ineligible via `tm['a'] <= 0 || tm['d'] <= 0` — a negative scale is a legitimate axis flip, not a degenerate one. Every block of such a document was dropped, so zero logical lines were built and the cross-block AND cross-line passes were silently disabled. Reject only a near-zero (degenerate) scale instead. 2. The cross-block v1 splice assumes the match starts at the LAST op of its block and ends at the FIRST op of the next (the tagged-PDF /Span shape). With one Tj per glyph a word match starts at op0 and ends at the last glyph — the opposite alignment — so it was skipped. Add a dedicated whole-block-consumption splice (buildWholeBlockSpanEdit) that replaces [startBlock.bt_offset, endBlock.et_offset+2] with a single fallback-font placeholder, re-emitting the start block's BT/Tm/Td setup (and any prefix glyph-ops) so the placeholder lands at the matched text. Trailing punctuation glued to the end word is consumed; a trailing letter/digit is not (so prefixes like "Jan" in "Janssen" are never over-redacted). The new path is gated on the per-glyph signature (a boundary block with >1 operator), so pure one-op-per-block spans keep using the unchanged v1 path. Verified on a Google-Docs PDF: 17 multi-word residuals -> 0 (the 2 remaining are 1-/3-char NER over-detections, "U" and "tel"); ReportLab and tagged-PDF paths unchanged.
These PDFs (Collabora/LibreOffice ODT->PDF export) previously produced "anonymised" output where nothing was replaced. Three independent bugs combined to make replace_text_in_document a silent no-op for this mainstream producer: 1. Operator boundary detection (findNextOperator / findNextAnyOperator) required whitespace on both sides of an operator token. LibreOffice packs operators against their operand delimiters (`/F1 20 Tf[<..>]TJ`), so every `]TJ` / `Tf[` was skipped and no text-showing operator was ever found. Per PDF 1.7 §7.2.2 a delimiter is equally a valid token boundary; accept closing delimiters ()>]}) before a token and any delimiter after it, while still excluding `/` so a `/TJ` name is not mistaken for an operator. 2. injectFallbackFontResource only handled inline dicts and PDFValueReference for /Resources and /Font. LibreOffice emits them as bare `N 0 R` references parsed as PDFValueSimple, which fell through and triggered method_exists(false). Add indirectReferenceOid() to resolve both reference shapes (mirroring the /Contents handling in buildFontContext()). 3. processTjArray bailed with cid_split_mismatch whenever a match boundary fell inside a multi-char hex fragment, leaving those entities unredacted. For single-byte-CMap fonts the raw bytes map 1:1 to the resolved text, so the fragment is byte-sliceable; drop the literal-only restriction (keep the strlen 1:1 guard) and preserve each fragment's shape into the re-emitted head/tail entries. Verified end-to-end against a Collabora-exported aanslagbiljet: all detected entities (person, BSN, street, e-mail, organisation) are now removed from the content stream and replaced via the registered Helvetica fallback font. Assisted-by: ClaudeCode:claude-opus-4-8 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the three no-op bugs found and fixed for Collabora/LibreOffice (ODT->PDF) output in the flagship text-replacement design doc: delimiter-packed operators (PC1), bare `N 0 R` Resources/Font references (PC2), and hex-fragment byte-slicing across a match boundary (PC3), plus end-to-end verification and the multi-byte CID follow-up note. Assisted-by: ClaudeCode:claude-opus-4-8 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Heropend na de migratie van Codeberg naar GitHub (2026-07-14).
Origineel, met review-historie: https://codeberg.org/Conduction/sapp/pulls/2