Skip to content

Make ExtractionDocument traversal complete and independent of section shape - #7963

Open
luisquintanilla wants to merge 3 commits into
dotnet:mainfrom
luisquintanilla:luisquintanilla-extraction-document-traversal
Open

luisquintanilla wants to merge 3 commits into
dotnet:mainfrom
luisquintanilla:luisquintanilla-extraction-document-traversal

Conversation

@luisquintanilla

@luisquintanilla luisquintanilla commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #7962

Summary

Make reading an ExtractionDocument complete and independent of how a provider shapes its sections. Today the obvious loop silently drops text, and how much it drops depends on the provider: on the same 10 PDFs it reads 76–83% of the words from flat providers but only 5–6% from the Azure providers. This PR adds GetText(), adds EnumerateContent() to sections, makes table-cell enumeration anchor-only, and documents where text can live.

// All text, whatever the provider or section shape.
string text = document.GetText();

// Per-element processing (filter roles, chunk, cite).
foreach (ExtractionDocumentElement element in document.EnumerateContent())
{
    string elementText = element.GetText(); // includes table cells and image text
}

The hierarchy is unchanged. This PR changes only Microsoft.Extensions.Documents.Abstractions, and every new API is [Experimental] like the rest of the package.

Why: two ways to lose text

The model from #7835 lets each producer choose the section shape. Some emit one flat section per page; others nest sections under headings, and table cells can themselves contain sections. Illustrative shapes:

Flat (one section per page)            Nested (sections follow headings)
Section                                Section
├─ Text "Revenue"                      ├─ Section
├─ Table                               │  ├─ Text "Revenue"
│  └─ Cell → Text "$42"                │  └─ Table
└─ Image (Text: "Q1 Q2 Q3")            │     └─ Cell → Section → Text "$42"
                                       └─ Image (Text: "Q1 Q2 Q3")

The loop most people write first:

foreach (var section in document.Sections)
{
    foreach (var element in section.Elements)
    {
        if (element is ExtractionDocumentTextElement { Content: TextContent text })
        {
            Console.WriteLine(text.Text);
        }
        // A nested ExtractionDocumentSection lands here and is skipped.
    }
}
  • Depth. It sees every text element of a flat document and skips every nested section of a nested one.
  • Type filtering. Even a depth-correct walk loses text if it keeps only ExtractionDocumentTextElement. Text also lives in table cells (whose content can itself be a section), in ExtractionDocumentImageElement.Text, and in the local Content of the document, sections and tables.

Before this PR, no public API returned "all the text" and the docs did not say where text can live, so every consumer had to write a correct recursive walker by hand. Our own docs and samples did not.

Measured with five providers on 10 hard pages from ParseBench-2. The number is the share of each page's ground-truth words a strategy returns:

Provider Obvious loop document.GetText()
Azure Content Understanding 0.05 0.92
Azure Document Intelligence 0.06 0.85
Mistral OCR 0.76 0.76
Ollama VLM 0.83 0.83
PdfPig 0.81 0.81

Flat providers don't change, so nothing regresses. Compare within a row; this isn't a provider ranking. The other strategies, per-page notes and how to reproduce are under Evidence below.

What this PR adds

API Change
ExtractionDocument.GetText() New. All text in document order.
ExtractionDocumentElement.GetText() New. All text in an element and its descendants.
ExtractionDocumentTableCell.GetText() New. All text in a cell's content.
protected virtual string? ExtractionDocumentElement.GetAdditionalText() New. Optional hook for custom element types: returns text stored outside the TextContent payload, which GetText() reads after the payload. The default returns null. ExtractionDocumentImageElement overrides it to return Text.
ExtractionDocumentSection.EnumerateContent() New. Same semantics as ExtractionDocument.EnumerateContent(), scoped to a section.
ExtractionDocumentTableCellCollection.GetCoveringCell(int, int) Now public (was internal). Resolves any grid position to its anchor.
ExtractionDocumentTableCellCollection : IEnumerable<ExtractionDocumentTableCell> Changed from IEnumerable<ExtractionDocumentTableCell?>. Enumeration yields anchor cells only, with no null placeholders for empty or covered positions. The indexer is unchanged.

The three GetText() methods are non-virtual instance methods, following the Framework Design Guidelines' preference for instance methods over extension methods on types you own (review feedback). They share one internal traversal, so every overload has the same semantics. Custom element types extend it through the protected virtual GetAdditionalText() hook rather than by overriding GetText(), so ordering, separators and cycle detection stay in one place. Overriding is optional: a custom element's TextContent payload is read without an override.

Docs:

  • The XML docs on Sections, Elements, EnumerateContent and ImageElement.Text now say that producers choose the section shape and where text can live.
  • The README gains a Reading a document section with the three supported patterns, and its example now uses GetText().

Reading a document after this PR

  1. All text: document.GetText().
  2. Per-element processing (filter roles, chunk, cite): foreach (var e in document.EnumerateContent()) and call e.GetText() on every element; don't keep only text elements.
  3. Structure-aware walks: recurse through Elements, table.Cells (anchors) and GetCoveringCell, as before.

Note

The obvious two-level loop above still compiles and is still lossy for nested providers. The hierarchy stays on purpose: flattening it would lose information that chunkers, citation and layout consumers need. This PR makes the correct path the easy one rather than making the wrong one impossible. It mirrors LINQ to XML (Elements() / Descendants() / Value). An analyzer that flags the loop is API question 4.

How this fits with the other PRs

Compatibility

  • Source: the Cells element type changes from nullable to non-nullable. Existing cell?.Content and if (cell is null) checks still compile; they become dead code. Code that relied on Cells.Count() == RowCount * ColumnCount breaks; use the indexer or GetCoveringCell. The package is experimental and preview.
  • Behavior: foreach (var cell in table.Cells) no longer yields null for empty or covered positions.
  • In-repo: IngestionDocumentElementExtensions.EnumerateSources and ImageAlternativeTextEnricher.EnumerateImages (Integrate shared Documents while retaining IngestionDocument #7904) already use cell?.Content, so they are unaffected.
  • CommunityToolkit/AI DocumentExtraction: 644/644 unit tests pass against this package, the same as against the published 11.0.0-preview.1.26560.3.

Suggested review focus

  1. GetText semantics (ExtractionDocumentText.cs): what's included, order, separator. The full rules are under GetText semantics below.
  2. Anchor-only Cells (ExtractionDocumentTable.cs): the one breaking change, plus GetCoveringCell becoming public.
  3. Docs (ExtractionDocument.cs, ExtractionDocumentElement.cs, README.md): do they make the shape-independent path obvious?
  4. Tests (DocumentTraversalTests.cs): especially the provider-shape test, which asserts that flat, nested and cell-nested shapes of the same content produce identical GetText().

Validation

  • New DocumentTraversalTests plus updated table tests: 239/239 pass on all TFMs.
  • API baseline (Microsoft.Extensions.Documents.Abstractions.json) regenerated.
  • CommunityToolkit/AI DocumentExtraction: 644/644 against this package.
  • Live: 5 providers × 10 ParseBench-2 pages, summarized above and detailed below.
  • Live A/B for the final commit (instance methods plus the GetAdditionalText hook) against the build before the hook, on the same 10 pages: all 174 per-document output files for PdfPig, Content Understanding and Document Intelligence are byte-identical, including 24 image elements whose text now goes through the hook. Mistral differs on 3 pages, all with no images and with different provider structure between calls (OCR run-to-run variation). In the final run, D == E on all 45 extracted documents across the 5 providers, and A == E for every flat provider.

Questions for API review

  1. Name of EnumerateContent. "Content" collides with the Content payload property, and the method doesn't return payloads. EnumerateLeaves or Descendants (with Descendants also yielding sections, as in LINQ to XML)?
  2. Separator. Environment.NewLine vs "\n" for deterministic output across operating systems (chunk hashes, snapshots).
  3. Image text. Include image.Text (OCR) and exclude AlternativeText (caption)? Or add an option or overload?
  4. Analyzer follow-up. Flag foreach … in document.Sections → foreach … in section.Elements with an is ExtractionDocumentTextElement filter and no recursion, and offer GetText() / EnumerateContent() as the fix?
  5. Cells enumeration break. Is anchor-only enumeration acceptable in preview, or should it be a new Anchors property with Cells left as is?
  6. ImageElement.Text vs. GetText(). The guidelines say not to have a property with the same name as a Get method, and image elements now have both, with different values: Text is the recognized text alone, while GetText() returns local TextContent followed by Text. Rename the property (for example RecognizedText), rename the method, or keep both and rely on the docs?
  7. Custom-element text hook. GetText() is non-virtual; custom elements contribute text through protected virtual string? GetAdditionalText(), read after the payload and before children. Overriding is optional, and adding the hook is non-breaking. Is the name right (GetAdditionalText, GetOwnText, GetLocalText)? And should the hook also cover custom containers, whose children neither GetText() nor EnumerateContent() traverse today?
Evidence: live measurements (5 providers × 10 ParseBench-2 pages)

Mean ground-truth token recall over 10 single-page PDFs from llamaindex/ParseBench-2: charts, complex table headers ×2, dense, financial, form, headings, layout, timetable and CJK pages. Every provider was created through the public samples repo, and each result was read five ways:

Provider Nested docs A: two-level loop B: EnumerateContent + is TextElement C: samples' walker (before fix) D: EnumerateContent + GetText() E: document.GetText()
Azure Content Understanding 10/10 0.05 0.48 0.84 0.92 0.92
Azure Document Intelligence 8/9 0.06 0.50 0.77 0.85 0.85
Mistral OCR 0/10 0.76 0.76 0.76 0.76 0.76
Ollama VLM 0/9 0.83 0.83 0.83 0.83 0.83
PdfPig 0/10 0.81 0.81 0.81 0.81 0.81
  • The issue reproduces at scale. On the same PDFs, the obvious loop scores 0.76–0.83 on flat providers and 0.05 on nested ones. On a 10-K page it scores 0.00 vs GetText() 0.99 for both Azure providers.
  • Fixing depth alone isn't enough. B handles nesting but drops table cells: on the timetable page B scores 0.02 vs 1.00 for E.
  • Cells alone aren't enough either. C drops chart labels stored in image text: on the charts page it scores 0.29 vs 1.00.
  • Nothing regresses. D == E on every row, and for flat providers every strategy agrees.
  • The remaining gap to 1.0 comes from extraction quality (a rotated page, CJK font decoding), not traversal.
  • Gaps in the run. Ollama timed out on one page. Document Intelligence fails on another because of a CommunityToolkit table bug (CommunityToolkit/AI#95).

Reproduce: the harness, pinned corpus and per-page numbers are in validation/parsebench (luisquintanilla/dotnet-document-extraction-samples#3). Running it writes each document's structure and the text every strategy read.

GetText semantics
  • Depth-first pre-order traversal; a node's local TextContent comes before its children.
  • Reads table cells through their anchors in row-major order, and includes image.Text and any custom element's GetAdditionalText(), after the element's payload.
  • Excludes AlternativeText, bytes/references, metadata and raw representations.
  • Does not filter by role, so page headers and footers are included.
  • Joins non-empty blocks with Environment.NewLine.
  • Repeats shared acyclic content at each occurrence, and throws InvalidOperationException on cycles or null entries, matching EnumerateContent.
Design: why not make the loop impossible?

The hierarchy is unchanged on purpose: flattening it would lose information that chunkers, citation and layout consumers need. So this PR makes the correct path the easy one rather than making the wrong one impossible, in the same way LINQ to XML offers Elements(), Descendants() and Value over one tree.

Alternatives considered and rejected:

  • A separate Subsections collection. Two collections to keep in sync, and the loop is still lossy.
  • Flat Elements with parent pointers. A large redesign that conflicts with shared nodes, and it still doesn't fix type filtering.
Review map (10 files, +648 / −71)

All paths are under src/Libraries/Microsoft.Extensions.Documents.Abstractions/ or test/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/.

File What changed
ExtractionDocumentText.cs New internal traversal shared by the three GetText() methods.
ExtractionDocumentElement.cs New ExtractionDocumentElement.GetText(), the protected virtual GetAdditionalText() hook (overridden by the image element) and ExtractionDocumentSection.EnumerateContent(); docs on Elements and ImageElement.Text.
ExtractionDocument.cs New GetText() with the full semantics docs; docs on Sections and EnumerateContent (section shape, where text lives); EnumerateSection becomes internal for reuse.
ExtractionDocumentTable.cs Anchor-only enumeration; public GetCoveringCell; new ExtractionDocumentTableCell.GetText().
README.md New Reading a document section; example uses GetText(); notes the custom-element hook.
Microsoft.Extensions.Documents.Abstractions.json Regenerated API baseline.
DocumentTraversalTests.cs New: section EnumerateContent (nesting, independence, cycles/nulls); anchor-only Cells plus GetCoveringCell over every position; GetText order across every text location, empty blocks, shared content, cycles, null entries; custom elements with and without the hook; provider-shape equivalence.
DocumentTableCellCollectionTests.cs, DocumentTableElementTests.cs Updated for anchor-only enumeration.
DocumentReferencesTests.cs API-surface test updated now that GetCoveringCell is public.
Microsoft Reviewers: Open in CodeFlow

Add GetText extensions for documents, elements and table cells that read
every text location (nested sections, table cells, image text and local
payloads) in document order. Add ExtractionDocumentSection.EnumerateContent,
make table cell enumeration yield only anchor cells, and make
GetCoveringCell public. Document that producers choose the section shape
and add a "Reading a document" README section.

Fixes dotnet#7962

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The README’s filtering example uses case-sensitive role matching, which can leave repeated headers and footers in the output when providers return different casing.

1 open issue
What changed in this PR

Addresses #7962 by adding complete text extraction to the shared Documents model, independent of provider section nesting, while preserving the existing hierarchy.

Changes:

  • Adds GetText() overloads for documents, elements and cells, including local payloads, table content and image text.
  • Adds section-level traversal, anchor-only cell enumeration and public merged-cell lookup.
  • Updates reading guidance, API baselines and tests for traversal and table behavior.
File Description
test/​Libraries/​Microsoft.Extensions.Documents.Abstractions.Tests/​DocumentTraversalTests.cs Adds traversal, text extraction and provider-shape tests.
test/​Libraries/​Microsoft.Extensions.Documents.Abstractions.Tests/​DocumentTableElementTests.cs Updates anchor-enumeration and cell-lookup assertions.
test/​Libraries/​Microsoft.Extensions.Documents.Abstractions.Tests/​DocumentTableCellCollectionTests.cs Verifies anchor ordering and merged-cell enumeration.
test/​Libraries/​Microsoft.Extensions.Documents.Abstractions.Tests/​DocumentReferencesTests.cs Checks the newly public cell-lookup API.
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​README.md Documents reading patterns and updates examples.
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​Microsoft.Extensions.Documents.Abstractions.json Records the experimental API changes.
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​ExtractionDocumentTable.cs Enumerates anchors only and exposes GetCoveringCell().
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​ExtractionDocumentExtensions.cs Implements recursive text extraction with cycle checks.
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​ExtractionDocumentElement.cs Adds section traversal and clarifies image-text documentation.
src/​Libraries/​Microsoft.Extensions.Documents.Abstractions/​ExtractionDocument.cs Clarifies traversal limits and shares the section walker.

Review effort: Balanced

@adamsitnik adamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luisquintanilla PTAL at my comment, thanks!

luisquintanilla and others added 2 commits November 20, 2026 12:13
Per review feedback and the Framework Design Guidelines, prefer instance methods
over extension methods on types we own. GetText is now a non-virtual instance
method on ExtractionDocument, ExtractionDocumentElement and
ExtractionDocumentTableCell; the shared traversal lives in an internal helper and
the public ExtractionDocumentExtensions class is removed. Call sites written as
x.GetText() compile unchanged.

Also make the README role-filter example compare roles case-insensitively, as the
README recommends.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
GetText() stays non-virtual. Custom element types can optionally override GetAdditionalText() to contribute text stored outside their TextContent payload; the walker keeps ordering, separators and cycle detection. Image Text is now read through the same hook, so output is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@luisquintanilla
luisquintanilla marked this pull request as ready for review November 20, 2026 19:45

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-ai Microsoft.Extensions.AI libraries

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[API Feedback]: ExtractionDocument traversal: the obvious loop silently drops content, and the result depends on the provider

3 participants