Skip to content

feat(replace): support chained [/ASCII85Decode /FlateDecode] filters - #11

Open
MWest2020 wants to merge 8 commits into
feat/cross-line-matchingfrom
feat/chained-filter-text-replace
Open

MWest2020 wants to merge 8 commits into
feat/cross-line-matchingfrom
feat/chained-filter-text-replace

Conversation

@MWest2020

Copy link
Copy Markdown

Heropend na de migratie van Codeberg naar GitHub (2026-07-14).
Origineel, met review-historie: https://codeberg.org/Conduction/sapp/pulls/2

rjzondervan and others added 8 commits June 16, 2026 10:05
replace_text_in_document only accepted a single literal /FlateDecode filter,
so content streams encoded with a chained filter array (e.g. ReportLab's
[/ASCII85Decode /FlateDecode]) were skipped entirely (streams_scanned=0),
leaving all entity text unredacted. Decode (apply_filter_chain_decode) and
re-encode (apply_filter_chain_encode, preserving /Filter shape) already
support these chains, so the gate is widened to any chain containing
/FlateDecode; unsupported chains remain fail-safe (skip on decode, no-op on
encode).

Also guard collectPageFonts against get_keys() returning false on
unparsable /Font dicts (was emitting foreach-on-false warnings).
…alueSimple

Some producers emit /Resources or /Font as a bare "6 0 R" token that the
parser stores as a PDFValueSimple rather than a PDFValueReference. The
is_a(PDFValueReference) gates in collectPageFonts missed that form, so the
font dict was never dereferenced (get_keys() === false) and the page's fonts
were not collected. With no font info, resolveOperandToUnicode fell back to a
raw-byte passthrough, leaving WinAnsi high bytes (e.g. 0xEB) unconverted — so
accented needles (ë/ï/ü) never matched and that text was left unredacted.

Add resolveIndirectValue() handling both reference forms (PDFValueSimple
get_object_referenced() already parses "N G R") and use it for both the
/Resources and /Font resolution. Fonts now resolve, accented text resolves to
UTF-8 via the font's WinAnsi encoding, and matching + splicing work via the
existing font-aware path.
parseContentStreamModel was the dominant cost of replace_text_in_document on
large PDFs — measured at 89% of replace time (67.8s of 75.9s on a 50-page doc,
1 needle), because its inner loop called findNextOperator() once per operator
TYPE (Tj/TJ/Tf/Tm) at every step. A type absent from the rest of a block scans
forward to its next occurrence anywhere ahead, so the per-stream parse is
O(n^2); it is also rebuilt 3x per stream (once per cross-phase).

Add findNextAnyOperator(): the same string-state-aware tokenizer, but it
returns the earliest operator of ANY requested type in a single forward scan,
so each step advances only to the next operator (linear per block). Behaviour
is identical — the old code already chose the earliest-position candidate.

Result: replace_text on the 50-page fixture drops 76s -> 12.5s (1 needle) and a
realistic 445-needle per-document run completes in ~14s (peak 14MB) where it
previously exceeded 2 minutes.
… parse

PDF comments (`%` to end-of-line) are whitespace-equivalent everywhere outside
strings and stream data, so they may appear between dictionary entries. The
value parser already skipped T_COMMENT, but the object/dict parser (_parse_obj)
did not — a comment inside a dictionary hit `default: throw "Invalid token"`.

ReportLab writes a `% ... -- digest (opensource)` line INSIDE the trailer
dictionary, which made get_trailer() report "trailer is not valid",
acquire_structure() fail, and PDFDoc::from_string() return false — so every
ReportLab-generated PDF (very common) failed to load and could not be
anonymised/text-replaced.

Fix centrally in nexttoken(): transparently skip T_COMMENT tokens so every
parse context (dictionaries, lists, values) tolerates comments. Strings,
hex-strings and streams use dedicated readers and are unaffected.

Verified: PDFDoc::from_string() now loads a ReportLab PDF that previously
returned false.
buildFontInfo() flags a font's /Encoding as "unhandled" only when it is an
inline dictionary (/BaseEncoding + /Differences). The detection used
`$enc->get_keys() !== []`, but get_keys() returns `false` (not `[]`) for a NAME
value (PDFValueType). Since `false !== []` is true, a *named* encoding such as
`/Identity-H` was wrongly flagged as an unhandled dict, fell back to WinAnsi,
and never used the /ToUnicode CMap — so CID (Type0/CIDFontType2/Identity-H)
fonts matched nothing and replaced zero text.

This is exactly the Google Docs / Skia ("AAAAAA+ArialMT" subset, Identity-H)
case: text extracts fine (ToUnicode) but replace_text_in_document produced
streams_modified=0 and encoding_dict_unhandled for the font.

Guard the dict check with is_array() so only a real inline dict is treated as
unhandled; named encodings route to FontEncoding::forName() and use the
ToUnicode CMap. Verified on a Google-Docs PDF: "Rotterdam" ×9 -> 0 (9
placeholders, streams_modified=2), encoding_dict_unhandled cleared.
…) PDFs

Skia / Google-Docs / PDF.js exports lay text out as one BT/ET block per
word with one Tj per glyph, positioned via Td under a vertical-flip text
matrix (d = -1). On these documents multi-word entities were never
replaced (single words were), leaving PII residuals.

Two compounding causes, both fixed:

1. groupBlocksIntoLogicalLines / groupBlocksIntoLinePairs rejected the
   blocks as ineligible via `tm['a'] <= 0 || tm['d'] <= 0` — a negative
   scale is a legitimate axis flip, not a degenerate one. Every block of
   such a document was dropped, so zero logical lines were built and the
   cross-block AND cross-line passes were silently disabled. Reject only
   a near-zero (degenerate) scale instead.

2. The cross-block v1 splice assumes the match starts at the LAST op of
   its block and ends at the FIRST op of the next (the tagged-PDF /Span
   shape). With one Tj per glyph a word match starts at op0 and ends at
   the last glyph — the opposite alignment — so it was skipped. Add a
   dedicated whole-block-consumption splice (buildWholeBlockSpanEdit)
   that replaces [startBlock.bt_offset, endBlock.et_offset+2] with a
   single fallback-font placeholder, re-emitting the start block's
   BT/Tm/Td setup (and any prefix glyph-ops) so the placeholder lands at
   the matched text. Trailing punctuation glued to the end word is
   consumed; a trailing letter/digit is not (so prefixes like "Jan" in
   "Janssen" are never over-redacted).

The new path is gated on the per-glyph signature (a boundary block with
>1 operator), so pure one-op-per-block spans keep using the unchanged v1
path. Verified on a Google-Docs PDF: 17 multi-word residuals -> 0 (the
2 remaining are 1-/3-char NER over-detections, "U" and "tel"); ReportLab
and tagged-PDF paths unchanged.
These PDFs (Collabora/LibreOffice ODT->PDF export) previously produced
"anonymised" output where nothing was replaced. Three independent bugs
combined to make replace_text_in_document a silent no-op for this
mainstream producer:

1. Operator boundary detection (findNextOperator / findNextAnyOperator)
   required whitespace on both sides of an operator token. LibreOffice
   packs operators against their operand delimiters (`/F1 20 Tf[<..>]TJ`),
   so every `]TJ` / `Tf[` was skipped and no text-showing operator was
   ever found. Per PDF 1.7 §7.2.2 a delimiter is equally a valid token
   boundary; accept closing delimiters ()>]}) before a token and any
   delimiter after it, while still excluding `/` so a `/TJ` name is not
   mistaken for an operator.

2. injectFallbackFontResource only handled inline dicts and
   PDFValueReference for /Resources and /Font. LibreOffice emits them as
   bare `N 0 R` references parsed as PDFValueSimple, which fell through
   and triggered method_exists(false). Add indirectReferenceOid() to
   resolve both reference shapes (mirroring the /Contents handling in
   buildFontContext()).

3. processTjArray bailed with cid_split_mismatch whenever a match
   boundary fell inside a multi-char hex fragment, leaving those entities
   unredacted. For single-byte-CMap fonts the raw bytes map 1:1 to the
   resolved text, so the fragment is byte-sliceable; drop the
   literal-only restriction (keep the strlen 1:1 guard) and preserve each
   fragment's shape into the re-emitted head/tail entries.

Verified end-to-end against a Collabora-exported aanslagbiljet: all
detected entities (person, BSN, street, e-mail, organisation) are now
removed from the content stream and replaced via the registered
Helvetica fallback font.

Assisted-by: ClaudeCode:claude-opus-4-8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record the three no-op bugs found and fixed for Collabora/LibreOffice
(ODT->PDF) output in the flagship text-replacement design doc:
delimiter-packed operators (PC1), bare `N 0 R` Resources/Font references
(PC2), and hex-fragment byte-slicing across a match boundary (PC3), plus
end-to-end verification and the multi-byte CID follow-up note.

Assisted-by: ClaudeCode:claude-opus-4-8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants