Skip to content

fix(markitdown-ocr): sort placeholders by length before substitution to prevent prefix collision - #2387

Open
Azhar (bunnysayzz) wants to merge 1 commit into
microsoft:mainfrom
bunnysayzz:fix/ocr-placeholder-prefix-collision-2383
Open

fix(markitdown-ocr): sort placeholders by length before substitution to prevent prefix collision#2387
Azhar (bunnysayzz) wants to merge 1 commit into
microsoft:mainfrom
bunnysayzz:fix/ocr-placeholder-prefix-collision-2383

Conversation

@bunnysayzz

Copy link
Copy Markdown

Root cause

_PLACEHOLDER = "MARKITDOWNOCRBLOCK{}" in packages/markitdown-ocr/src/markitdown_ocr/_docx_converter_with_ocr.py.

The substitution loop in DocxConverterWithOCR._convert_local() iterates in ascending index order and calls str.replace(placeholder, ocr_block) for each image. Because "MARKITDOWNOCRBLOCK1" is a prefix of "MARKITDOWNOCRBLOCK10" through "MARKITDOWNOCRBLOCK19", replacing index 1 first corrupts every two-digit placeholder: MARKITDOWNOCRBLOCK10 becomes the OCR content of block 1 followed by a literal "0", dropping the OCR content for image 10 and duplicating the content for image 1.

Reproduction (no OCR service needed):

_PLACEHOLDER = "MARKITDOWNOCRBLOCK{}"
md = " ".join(f"<p>{_PLACEHOLDER.format(i)}</p>" for i in range(11))
for i in range(11):
    md = md.replace(_PLACEHOLDER.format(i), f"[OCR:{i}]")
# Result: ... <p>[OCR:1]</p> ... <p>[OCR:1]0</p>
# OCR block 10 is gone; block 1 appears twice.

Fix

Build the full list of (placeholder, ocr_block) pairs first, then sort by descending placeholder length before iterating. Longer placeholders (MARKITDOWNOCRBLOCK10) are replaced before shorter ones (MARKITDOWNOCRBLOCK1), so no prefix collision can occur.

replacements = [
    (_PLACEHOLDER.format(i), f"*[Image OCR]\n{raw_text}\n[End OCR]*")
    for i, raw_text in enumerate(ocr_texts)
]
replacements.sort(key=lambda pair: len(pair[0]), reverse=True)
for placeholder, ocr_block in replacements:
    md = md.replace(placeholder, ocr_block)

Test evidence

Five regression tests in packages/markitdown-ocr/tests/test_ocr_placeholder_collision.py:

PASSED tests/test_ocr_placeholder_collision.py::test_buggy_code_leaves_corrupted_placeholder_for_11_images
PASSED tests/test_ocr_placeholder_collision.py::test_fixed_code_no_remaining_placeholders_for_11_images
PASSED tests/test_ocr_placeholder_collision.py::test_fixed_code_all_ocr_blocks_present_for_11_images
PASSED tests/test_ocr_placeholder_collision.py::test_fixed_code_no_remaining_placeholders_for_100_images
PASSED tests/test_ocr_placeholder_collision.py::test_no_regression_under_10_images
5 passed in 0.02s

The first test (test_buggy_code_leaves_corrupted_placeholder_for_11_images) documents the bug by asserting the old behavior. The remaining four verify the fix.

Fixes #2383

@bunnysayzz

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

…to prevent prefix collision

When a DOCX has 11 or more images, the substitution loop in
DocxConverterWithOCR._convert_local() iterates in ascending index order and
calls str.replace(placeholder, ocr_block) for each. Because
"MARKITDOWNOCRBLOCK1" is a prefix of "MARKITDOWNOCRBLOCK10" through
"MARKITDOWNOCRBLOCK19", replacing index 1 first corrupts every two-digit
placeholder: MARKITDOWNOCRBLOCK10 becomes the content of block 1 followed by
a literal "0", dropping the OCR content for image 10 and duplicating the
content for image 1.

Fix: build the full list of (placeholder, ocr_block) pairs first, then sort
by descending placeholder length before iterating. Longer placeholders
(MARKITDOWNOCRBLOCK10) are replaced before shorter ones (MARKITDOWNOCRBLOCK1),
so no prefix collision can occur.

Five regression tests added to tests/test_ocr_placeholder_collision.py:
- test_buggy_code_leaves_corrupted_placeholder_for_11_images: documents the bug
- test_fixed_code_no_remaining_placeholders_for_11_images: verifies fix (11 images)
- test_fixed_code_all_ocr_blocks_present_for_11_images: all blocks present
- test_fixed_code_no_remaining_placeholders_for_100_images: verifies fix (100 images)
- test_no_regression_under_10_images: no regression for small documents

Fixes microsoft#2383
@bunnysayzz
Azhar (bunnysayzz) force-pushed the fix/ocr-placeholder-prefix-collision-2383 branch from da02680 to 1808375 Compare September 8, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

markitdown-ocr: placeholder prefix collision drops OCR blocks in DOCX with 11 or more images

1 participant