Repository navigation
refactor(parse): write an image as Azure Document Intelligence writes a figure - #807
Merged
Merged
Conversation
… a figure
The *[Image OCR p:N]* / *[End OCR]* lines predate the move to the shared
Microsoft.Extensions.Documents tree. They claimed to be MarkItDown's markers but
were not (MarkItDown writes "*[Image OCR]" ... "[End OCR]*", one emphasis span),
and they put the page into the Markdown, which the Markdown-first contract
forbids.
What Microsoft provides, checked layer by layer:
- Microsoft.Extensions.Documents: ExtractionDocumentImageElement has Text (OCR or
transcription) and AlternativeText ("captions and descriptions"). Parse already
used Text; captions went into a sibling paragraph instead of AlternativeText.
- Microsoft.Extensions.DataIngestion (and dotnet/extensions#7904): images feed
chunks as AlternativeText ?? Text; no Markdown form for OCR text. The Markdig
reader maps only .
- Microsoft.Extensions.DocumentExtraction and Microsoft.Extensions.AI: nothing for
figures.
- Azure Document Intelligence's layout Markdown (GA, documented): a figure is a
<figure> block, its caption a <figcaption>, the text read from it inside. Vault
already ingests that format from its Azure engine, but kept it as raw HTML, so
Azure figures were invisible to classification and segmentation.
So the tree follows the shared model and the Markdown follows Azure DI:
- FigureMarkup replaces ImageOcrMarkup: <figure>, optional <figcaption> (HTML-
encoded, one line), blank line, the transcription, blank line, </figure>. The
blank lines keep the transcription Markdown in CommonMark. Tag lines in a
document's own text are backslash-escaped.
- The renderer writes image.AlternativeText as the figcaption; the reader turns
any <figure> (Vault's or Azure's) into an image with Text and AlternativeText.
- PDF caption lines and DOCX/PPTX alt text become AlternativeText. The page
anchor goes: the image's page is in its source references, and PPTX slides
already give it.
- MarkdownDocumentReader.ReadInto no longer takes the document (it only served
the page anchor).
- MarkdownSlicer reads a span's figure-ness from the Markdown line it cut at:
the LLM sees "<figure>" through PromptBoundary and may return it encoded.
- Classification and segmentation prompts describe the figure as the LLM reads
it. The evaluation scorer's tag stripping already covers the tags; the
baseline does not move.
Documents parsed before this keep the old lines; a re-parse rewrites them
(CHANGELOG deploy note).
Two problems from reviewing the <figure> switch:
- Every figure opened with the same "<figure>" line. Segmentation finds a
span by its first line, searching forward, and the LLM may fold a
decorative figure (a seal) into the text before it; the cut meant for the
next figure then landed on the folded one, shortening the text span and
mixing the seal into the figure child. The old p:N anchor kept PDF figures
on different pages apart. The renderer now numbers figures with the HTML id
attribute (<figure id="fig-1">, in reading order), as Marker
(data-original-image-id) and Azure's JSON (figures[].id) identify figures;
Azure's id-less form is still read. The prompts show the numbered form.
- Tag lines were matched after trimming, anywhere. An HTML sample in a code
block ("<figure>" on its own line is common there) was split into a fake
figure on read, set off segmentation, and got a visible backslash on write.
Tags now count only where CommonMark reads an HTML block: indented three
spaces or fewer, outside a fenced code block. A transcription that leaves a
fence open is closed inside its figure. The slicer asks OpensFigureAt, which
knows the fence state. A figure in a DOCX footnote is written after the note
instead of indented into it, where it would now be the note's text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
*[Image OCR p:N]*/*[End OCR]*figure markers predate the move to the sharedMicrosoft.Extensions.Documentstree. This replaces them with Microsoft's own forms: the tree's image element for the structure, and Azure Document Intelligence's<figure>block for the Markdown.What Microsoft provides
Microsoft.Extensions.Documents.AbstractionsExtractionDocumentImageElement.Text(OCR or transcription) andAlternativeText("captions and descriptions"). Parse usedText; captions went into a sibling paragraph.Microsoft.Extensions.DataIngestion(+ dotnet/extensions#7904)AlternativeText ?? Text. No Markdown form for OCR text; the Markdig reader maps only.Microsoft.Extensions.DocumentExtraction,Microsoft.Extensions.AI<figure>, caption in<figcaption>, the text read from the figure inside. Vault's Azure engine already returned this, but it stayed raw HTML, so classification and segmentation never saw Azure's figures.markitdown-ocr*[Image OCR]…[End OCR]*(one emphasis span). The old Vault markers claimed to match it and did not.Change
FigureMarkupreplacesImageOcrMarkup:<figure>, optional<figcaption>(HTML-encoded, one line), blank line, transcription, blank line,</figure>. The blank lines keep the transcription Markdown in CommonMark; tag lines in a document's own text are backslash-escaped.AlternativeTextas the caption;MarkdownDocumentReaderturns any<figure>(Vault's or Azure's) into an image element.AlternativeText. The page anchor is gone: page metadata in the Markdown string was already forbidden by the Markdown-first contract, and the image's page is in its source references.MarkdownSlicerreads a span's figure-ness from the line it cut at, because the LLM sees<figure>(PromptBoundary) and may return it encoded (mutation-tested).Document.Markdownconsumers; CHANGELOG has the entry and a deploy note (re-parse documents parsed before).Verification
<figure>boundary → figure slice; a<figure>chunked by Index keeps caption and table, no tags.