Skip to content

refactor(parse): write an image as Azure Document Intelligence writes a figure - #807

Merged
duguankui merged 2 commits into
mainfrom
refactor/figure-markup-azure-di
Nov 19, 2026
Merged

duguankui merged 2 commits into
mainfrom
refactor/figure-markup-azure-di

Conversation

@duguankui

Copy link
Copy Markdown
Member

The *[Image OCR p:N]* / *[End OCR]* figure markers predate the move to the shared Microsoft.Extensions.Documents tree. This replaces them with Microsoft's own forms: the tree's image element for the structure, and Azure Document Intelligence's <figure> block for the Markdown.

What Microsoft provides

Package / service Figure / OCR support
Microsoft.Extensions.Documents.Abstractions ExtractionDocumentImageElement.Text (OCR or transcription) and AlternativeText ("captions and descriptions"). Parse used Text; captions went into a sibling paragraph.
Microsoft.Extensions.DataIngestion (+ dotnet/extensions#7904) Images feed chunks as AlternativeText ?? Text. No Markdown form for OCR text; the Markdig reader maps only ![alt](url).
Microsoft.Extensions.DocumentExtraction, Microsoft.Extensions.AI Nothing for figures.
Azure Document Intelligence layout Markdown (GA, documented) <figure>, caption in <figcaption>, the text read from the figure inside. Vault's Azure engine already returned this, but it stayed raw HTML, so classification and segmentation never saw Azure's figures.
MarkItDown markitdown-ocr *[Image OCR] … [End OCR]* (one emphasis span). The old Vault markers claimed to match it and did not.

Change

  • FigureMarkup replaces ImageOcrMarkup: <figure>, optional <figcaption> (HTML-encoded, one line), blank line, transcription, blank line, </figure>. The blank lines keep the transcription Markdown in CommonMark; tag lines in a document's own text are backslash-escaped.
  • Renderer writes AlternativeText as the caption; MarkdownDocumentReader turns any <figure> (Vault's or Azure's) into an image element.
  • PDF caption lines and DOCX/PPTX alt text become AlternativeText. The page anchor is gone: page metadata in the Markdown string was already forbidden by the Markdown-first contract, and the image's page is in its source references.
  • MarkdownSlicer reads a span's figure-ness from the line it cut at, because the LLM sees &lt;figure> (PromptBoundary) and may return it encoded (mutation-tested).
  • Prompts describe the figure as the LLM reads it.
  • Breaking for Document.Markdown consumers; CHANGELOG has the entry and a deploy note (re-parse documents parsed before).

Verification

  • Full solution builds; Parse (incl. evaluation baseline, unchanged), Extract (Application, Domain, EF, HttpApi, Mcp) and Index EF tests pass locally.
  • New tests: Azure DI figure → image with caption; encoded &lt;figure> boundary → figure slice; a <figure> chunked by Index keeps caption and table, no tags.
  • Not verified: LLM behaviour of classification/segmentation on the new prompt wording against a live model.

… a figure

The *[Image OCR p:N]* / *[End OCR]* lines predate the move to the shared
Microsoft.Extensions.Documents tree. They claimed to be MarkItDown's markers but
were not (MarkItDown writes "*[Image OCR]" ... "[End OCR]*", one emphasis span),
and they put the page into the Markdown, which the Markdown-first contract
forbids.

What Microsoft provides, checked layer by layer:
- Microsoft.Extensions.Documents: ExtractionDocumentImageElement has Text (OCR or
  transcription) and AlternativeText ("captions and descriptions"). Parse already
  used Text; captions went into a sibling paragraph instead of AlternativeText.
- Microsoft.Extensions.DataIngestion (and dotnet/extensions#7904): images feed
  chunks as AlternativeText ?? Text; no Markdown form for OCR text. The Markdig
  reader maps only ![alt](url).
- Microsoft.Extensions.DocumentExtraction and Microsoft.Extensions.AI: nothing for
  figures.
- Azure Document Intelligence's layout Markdown (GA, documented): a figure is a
  <figure> block, its caption a <figcaption>, the text read from it inside. Vault
  already ingests that format from its Azure engine, but kept it as raw HTML, so
  Azure figures were invisible to classification and segmentation.

So the tree follows the shared model and the Markdown follows Azure DI:
- FigureMarkup replaces ImageOcrMarkup: <figure>, optional <figcaption> (HTML-
  encoded, one line), blank line, the transcription, blank line, </figure>. The
  blank lines keep the transcription Markdown in CommonMark. Tag lines in a
  document's own text are backslash-escaped.
- The renderer writes image.AlternativeText as the figcaption; the reader turns
  any <figure> (Vault's or Azure's) into an image with Text and AlternativeText.
- PDF caption lines and DOCX/PPTX alt text become AlternativeText. The page
  anchor goes: the image's page is in its source references, and PPTX slides
  already give it.
- MarkdownDocumentReader.ReadInto no longer takes the document (it only served
  the page anchor).
- MarkdownSlicer reads a span's figure-ness from the Markdown line it cut at:
  the LLM sees "&lt;figure>" through PromptBoundary and may return it encoded.
- Classification and segmentation prompts describe the figure as the LLM reads
  it. The evaluation scorer's tag stripping already covers the tags; the
  baseline does not move.

Documents parsed before this keep the old lines; a re-parse rewrites them
(CHANGELOG deploy note).
Two problems from reviewing the <figure> switch:

- Every figure opened with the same "<figure>" line. Segmentation finds a
  span by its first line, searching forward, and the LLM may fold a
  decorative figure (a seal) into the text before it; the cut meant for the
  next figure then landed on the folded one, shortening the text span and
  mixing the seal into the figure child. The old p:N anchor kept PDF figures
  on different pages apart. The renderer now numbers figures with the HTML id
  attribute (<figure id="fig-1">, in reading order), as Marker
  (data-original-image-id) and Azure's JSON (figures[].id) identify figures;
  Azure's id-less form is still read. The prompts show the numbered form.

- Tag lines were matched after trimming, anywhere. An HTML sample in a code
  block ("<figure>" on its own line is common there) was split into a fake
  figure on read, set off segmentation, and got a visible backslash on write.
  Tags now count only where CommonMark reads an HTML block: indented three
  spaces or fewer, outside a fenced code block. A transcription that leaves a
  fence open is closed inside its figure. The slicer asks OpensFigureAt, which
  knows the fence state. A figure in a DOCX footnote is written after the note
  instead of indented into it, where it would now be the note's text.
@duguankui
duguankui merged commit a043d2c into main Nov 19, 2026
2 checks passed
@duguankui
duguankui deleted the refactor/figure-markup-azure-di branch November 19, 2026 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant