Repository navigation
PoC: text replacement on FlateDecode content streams #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,4 @@ | ||
| /composer.lock | ||
| /vendor | ||
| /.idea/ | ||
| examples/poc-output.pdf |
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| <?php | ||
| /** | ||
| * PoC fixture generator for the text-replacement work on | ||
| * `work/text-replacement`. Produces a minimal one-page PDF that satisfies | ||
| * the PoC scope locked in the OR-side `pdf-anonymisation` change (design | ||
| * D4): | ||
| * | ||
| * - one FlateDecode-compressed content stream | ||
| * - one Helvetica font (PDF base font — no embedding required, simple | ||
| * /WinAnsiEncoding) | ||
| * - one Tj operator carrying the needle "Jan Jansen" in a body line | ||
| * | ||
| * No Identity-H, no /ToUnicode CMap, no TJ kerning arrays, no filter | ||
| * chaining. Those are deferred to PRs #06/#07 in `docs/upstream-prs/`. | ||
| * | ||
| * The output is written to `examples/poc-fixture.pdf`. Re-run after any | ||
| * fixture-shape change. | ||
| * | ||
| * Usage: | ||
| * php examples/poc-make-fixture.php | ||
| * | ||
| * Compatibility: targets PHP 7.4+. Avoids named arguments (8.0+) and | ||
| * other 8.x-only syntax so the fixture remains regeneratable on the | ||
| * project's declared minimum-supported PHP version. | ||
| * | ||
| * SPDX-License-Identifier: LGPL-3.0-or-later | ||
| */ | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| /** | ||
| * Build the page content stream that places the needle text on the page. | ||
| * | ||
| * Uses absolute PDF text-object operators (BT/ET, Tf, Td, Tj) per | ||
| * PDF 1.7 §9.4.1. WinAnsi font means ASCII bytes pass through literally | ||
| * — the body of the Tj string IS the searchable byte sequence. | ||
| * | ||
| * @return string The uncompressed content-stream bytes. | ||
| */ | ||
| function poc_build_content_stream(): string | ||
| { | ||
| $body = "Aanvraag van Jan Jansen voor het loket."; | ||
| return "BT\n/F1 12 Tf\n72 720 Td\n(" | ||
| . $body | ||
| . ") Tj\nET\n"; | ||
| } | ||
|
|
||
|
|
||
| /** | ||
| * Wrap the given object body in the standard `oid 0 obj ... endobj` | ||
| * syntax. Returns the bytes ready to be concatenated into the PDF. | ||
| * | ||
| * @param int $oid Object id. | ||
| * @param string $body Object body (dictionary, stream, etc.) — already serialised. | ||
| * | ||
| * @return string | ||
| */ | ||
| function poc_obj(int $oid, string $body): string | ||
| { | ||
| return $oid . " 0 obj\n" . $body . "\nendobj\n"; | ||
| } | ||
|
|
||
|
|
||
| $contentStream = poc_build_content_stream(); | ||
|
|
||
| // Pin the deflate level to 6 so the fixture is byte-identical across | ||
| // PHP/zlib versions (zlib's `Z_DEFAULT_COMPRESSION` is -1 which maps | ||
| // to different defaults depending on the linked zlib build). | ||
| $compressedStream = gzcompress($contentStream, 6); | ||
| if ($compressedStream === false) { | ||
| fwrite(STDERR, "FATAL: gzcompress failed on the content stream\n"); | ||
| exit(1); | ||
| } | ||
|
|
||
|
rjzondervan marked this conversation as resolved.
Resolved
rjzondervan marked this conversation as resolved.
Resolved
|
||
| // Object table — PDF 1.4 minimal viable shape: | ||
| // 1 /Catalog → 2 /Pages → 3 /Page → 4 /Contents stream | ||
| // 5 /Font (Helvetica, base font — no descriptor, no widths needed | ||
| // for a standard base font in PDF readers). | ||
| $obj1 = poc_obj(1, "<<\n /Type /Catalog\n /Pages 2 0 R\n>>"); | ||
|
|
||
| $obj2 = poc_obj(2, "<<\n /Type /Pages\n /Kids [ 3 0 R ]\n /Count 1\n>>"); | ||
|
|
||
| $obj3 = poc_obj( | ||
| 3, | ||
| "<<\n /Type /Page\n /Parent 2 0 R\n" | ||
| . " /MediaBox [ 0 0 612 792 ]\n" | ||
| . " /Resources <<\n /Font << /F1 5 0 R >>\n /ProcSet [ /PDF /Text ]\n >>\n" | ||
| . " /Contents 4 0 R\n>>" | ||
| ); | ||
|
|
||
| $obj4 = poc_obj( | ||
| 4, | ||
| "<<\n /Length " . strlen($compressedStream) . "\n /Filter /FlateDecode\n>>\n" | ||
| . "stream\n" . $compressedStream . "\nendstream" | ||
| ); | ||
|
|
||
| $obj5 = poc_obj( | ||
| 5, | ||
| "<<\n /Type /Font\n /Subtype /Type1\n /BaseFont /Helvetica\n /Encoding /WinAnsiEncoding\n>>" | ||
| ); | ||
|
|
||
| // Assemble. Track byte offsets for the xref table. | ||
| $pdf = "%PDF-1.4\n%\xE2\xE3\xCF\xD3\n"; | ||
| $offsets = []; | ||
| foreach ([1 => $obj1, 2 => $obj2, 3 => $obj3, 4 => $obj4, 5 => $obj5] as $oid => $body) { | ||
| $offsets[$oid] = strlen($pdf); | ||
| $pdf .= $body; | ||
| } | ||
|
|
||
| // xref table — 6 entries (object 0 is the head of the free-list). | ||
| $xrefStart = strlen($pdf); | ||
| $pdf .= "xref\n0 6\n"; | ||
| $pdf .= "0000000000 65535 f \n"; | ||
| foreach ([1, 2, 3, 4, 5] as $oid) { | ||
| $pdf .= sprintf("%010d 00000 n \n", $offsets[$oid]); | ||
| } | ||
|
|
||
| // Trailer. | ||
| $pdf .= "trailer\n<<\n /Size 6\n /Root 1 0 R\n>>\nstartxref\n" . $xrefStart . "\n%%EOF\n"; | ||
|
|
||
| $out = __DIR__ . '/poc-fixture.pdf'; | ||
| file_put_contents($out, $pdf); | ||
| echo "Wrote $out (" . strlen($pdf) . " bytes)\n"; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| <?php | ||
| /** | ||
| * PoC verify script for the `work/text-replacement` integration branch. | ||
| * | ||
| * Exercises the full pipeline end-to-end on the synthesised fixture: | ||
| * | ||
| * poc-fixture.pdf → PDFDoc::from_string | ||
| * → replace_text_in_document(['Jan Jansen' => '[PERSON: 7]', | ||
| * 'NotInDoc' => '[X]']) | ||
| * → to_pdf_file_s(true) | ||
| * → PDFDoc::from_string (reload) | ||
| * → exhaustive object-stream sweep | ||
| * → assert: no residual "Jan Jansen", presence of | ||
| * "[PERSON: 7]", `unmatched_needles == ['NotInDoc']`, | ||
| * `replacements_per_needle['NotInDoc'] == 0` | ||
| * | ||
| * The Phase-2 sweep is intentionally MORE PARANOID than the feature | ||
| * under test: it walks every object in the reloaded document and | ||
| * opportunistically gzuncompresses any binary-looking stream, then | ||
| * counts byte occurrences of both needle and placeholder. This is the | ||
| * test-of-tests safeguard — it does NOT replicate the production | ||
| * method's filter / BT skip heuristic, because a regression that | ||
| * tightens those skips would otherwise pass vacuously. | ||
| * | ||
| * Locks the consumer-side OpenRegister `pdf-anonymisation` design D5 | ||
| * (validation gate): output MUST NOT contain the needle in ANY | ||
| * decoded content stream after the round-trip. | ||
| * | ||
| * Usage: | ||
| * php examples/poc-replace-text.php | ||
| * | ||
| * SPDX-License-Identifier: LGPL-3.0-or-later | ||
| */ | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| require_once __DIR__ . '/../vendor/autoload.php'; | ||
|
|
||
| use ddn\sapp\PDFDoc; | ||
|
|
||
| $inputPath = __DIR__ . '/poc-fixture.pdf'; | ||
| $outputPath = __DIR__ . '/poc-output.pdf'; | ||
| $needle = 'Jan Jansen'; | ||
| $replacement = '[PERSON: 7]'; | ||
| $unmatchedNeedle = 'NotInDoc'; | ||
| $unmatchedReplacement = '[X]'; | ||
|
|
||
| if (file_exists($inputPath) === false) { | ||
| fwrite(STDERR, "FATAL: fixture not found at $inputPath — run examples/poc-make-fixture.php first.\n"); | ||
| exit(1); | ||
| } | ||
|
|
||
| echo "PoC: text-replacement pipeline verification\n"; | ||
| echo " fixture: $inputPath\n"; | ||
| echo " output: $outputPath\n"; | ||
| echo " needle: '$needle'\n"; | ||
| echo " placeholder:'$replacement'\n\n"; | ||
|
|
||
| // ----- Phase 1: decode → replace → re-encode ----- | ||
| $inputBytes = file_get_contents($inputPath); | ||
| $doc = PDFDoc::from_string($inputBytes); | ||
| if ($doc === false) { | ||
| fwrite(STDERR, "FATAL: PDFDoc::from_string failed on the fixture.\n"); | ||
| exit(1); | ||
| } | ||
|
|
||
| // Pass both a matching needle and an intentionally-unmatched one so | ||
| // the diagnostic surface's `unmatched_needles` accumulator is | ||
| // exercised — without this a regression that always zeroes the | ||
| // unmatched list would pass. | ||
| $stats = $doc->replace_text_in_document([ | ||
| $needle => $replacement, | ||
| $unmatchedNeedle => $unmatchedReplacement, | ||
| ]); | ||
|
|
||
| echo "Phase 1 — replace_text_in_document() returned:\n"; | ||
| echo " streams_scanned: {$stats['streams_scanned']}\n"; | ||
| echo " content_streams_scanned: {$stats['content_streams_scanned']}\n"; | ||
| echo " streams_modified: {$stats['streams_modified']}\n"; | ||
| echo " replacements_per_needle: " . json_encode($stats['replacements_per_needle']) . "\n"; | ||
| echo " unmatched_needles: " . json_encode($stats['unmatched_needles']) . "\n\n"; | ||
|
|
||
| // Per the method's docblock, `rebuild = true` is the supported | ||
| // serialisation path after `replace_text_in_document` populates | ||
| // `_pdf_objects`. The incremental path produces a broken PDF. | ||
| $serialised = $doc->to_pdf_file_s(true); | ||
| if ($serialised === false) { | ||
| fwrite(STDERR, "FATAL: to_pdf_file_s returned false.\n"); | ||
| exit(1); | ||
| } | ||
| file_put_contents($outputPath, $serialised); | ||
| echo "Wrote " . strlen($serialised) . " bytes to $outputPath\n\n"; | ||
|
|
||
| // ----- Phase 2: reload + EXHAUSTIVE object-stream sweep ----- | ||
| // | ||
| // Walks every object's raw stream (gzuncompressing opportunistically) | ||
| // and counts needle / placeholder occurrences. Does NOT replicate the | ||
| // production method's filter / BT-token skip heuristics — a feature | ||
| // regression that incorrectly skips a stream class would otherwise | ||
| // pass vacuously here. The sweep is permissive: any object whose raw | ||
| // stream is gzuncompressable contributes; non-decompressable streams | ||
| // are scanned for the needle as raw bytes as a defence in depth. | ||
| $reloaded = PDFDoc::from_string($serialised); | ||
| if ($reloaded === false) { | ||
| fwrite(STDERR, "FATAL: PDFDoc::from_string failed on the output PDF (round-trip broke).\n"); | ||
| exit(1); | ||
| } | ||
|
|
||
| $residualHits = 0; | ||
| $placeholderHits = 0; | ||
| $streamsScanned = 0; | ||
| foreach ($reloaded->get_object_iterator() as $oid => $obj) { | ||
| $raw = $obj->get_stream(true); | ||
| if (!is_string($raw) || $raw === '') { | ||
| continue; | ||
|
rjzondervan marked this conversation as resolved.
Resolved
|
||
| } | ||
| $streamsScanned++; | ||
|
|
||
| // Try gzuncompress (suppress warning on non-compressed streams). | ||
| $candidate = @gzuncompress($raw); | ||
| if ($candidate === false) { | ||
| $candidate = $raw; | ||
| } | ||
|
|
||
| $residualHits += substr_count($candidate, $needle); | ||
| $placeholderHits += substr_count($candidate, $replacement); | ||
| } | ||
|
|
||
| echo "Phase 2 — exhaustive re-extract sweep over reloaded output:\n"; | ||
| echo " streams_scanned: $streamsScanned (all object streams, not just /FlateDecode + BT)\n"; | ||
| echo " residual_needles: $residualHits ('$needle')\n"; | ||
| echo " placeholder_hits: $placeholderHits ('$replacement')\n\n"; | ||
|
|
||
| // ----- Phase 3: assertions ----- | ||
| $failures = []; | ||
| if ($stats['streams_modified'] < 1) { | ||
| $failures[] = 'expected at least 1 stream_modified, got ' . $stats['streams_modified']; | ||
| } | ||
| if (($stats['replacements_per_needle'][$needle] ?? 0) !== 1) { | ||
| $failures[] = "expected replacements_per_needle['$needle'] == 1, got " | ||
|
rjzondervan marked this conversation as resolved.
Resolved
|
||
| . ($stats['replacements_per_needle'][$needle] ?? 'missing'); | ||
| } | ||
| if (($stats['replacements_per_needle'][$unmatchedNeedle] ?? null) !== 0) { | ||
| $failures[] = "expected replacements_per_needle['$unmatchedNeedle'] == 0, got " | ||
| . var_export($stats['replacements_per_needle'][$unmatchedNeedle] ?? 'missing', true); | ||
| } | ||
| if (!in_array($unmatchedNeedle, $stats['unmatched_needles'], true)) { | ||
| $failures[] = "expected '$unmatchedNeedle' in unmatched_needles, got " | ||
| . json_encode($stats['unmatched_needles']); | ||
| } | ||
| if (in_array($needle, $stats['unmatched_needles'], true)) { | ||
| $failures[] = "did NOT expect '$needle' in unmatched_needles"; | ||
| } | ||
| if ($residualHits !== 0) { | ||
| $failures[] = "expected 0 residual '$needle' after round-trip, got $residualHits"; | ||
| } | ||
| if ($placeholderHits < 1) { | ||
| $failures[] = "expected at least 1 '$replacement' after round-trip, got $placeholderHits"; | ||
| } | ||
|
|
||
| if (count($failures) === 0) { | ||
| echo "RESULT: VERIFIED — text-replacement pipeline produces a clean redacted output.\n"; | ||
| exit(0); | ||
| } | ||
|
|
||
| echo "RESULT: FAIL — " . count($failures) . " assertion(s) violated:\n"; | ||
| foreach ($failures as $msg) { | ||
| echo " - $msg\n"; | ||
| } | ||
| exit(1); | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.