Repository navigation
Make ExtractionDocument traversal complete and independent of section shape - #7963
Open
luisquintanilla wants to merge 3 commits into
Open
luisquintanilla wants to merge 3 commits into
luisquintanilla wants to merge 3 commits into
Conversation
Add GetText extensions for documents, elements and table cells that read every text location (nested sections, table cells, image text and local payloads) in document order. Add ExtractionDocumentSection.EnumerateContent, make table cell enumeration yield only anchor cells, and make GetCoveringCell public. Document that producers choose the section shape and add a "Reading a document" README section. Fixes dotnet#7962 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The README’s filtering example uses case-sensitive role matching, which can leave repeated headers and footers in the output when providers return different casing.
1 open issue
What changed in this PR
Addresses #7962 by adding complete text extraction to the shared Documents model, independent of provider section nesting, while preserving the existing hierarchy.
Changes:
- Adds
GetText()overloads for documents, elements and cells, including local payloads, table content and image text. - Adds section-level traversal, anchor-only cell enumeration and public merged-cell lookup.
- Updates reading guidance, API baselines and tests for traversal and table behavior.
| File | Description |
|---|---|
test/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/DocumentTraversalTests.cs |
Adds traversal, text extraction and provider-shape tests. |
test/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/DocumentTableElementTests.cs |
Updates anchor-enumeration and cell-lookup assertions. |
test/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/DocumentTableCellCollectionTests.cs |
Verifies anchor ordering and merged-cell enumeration. |
test/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/DocumentReferencesTests.cs |
Checks the newly public cell-lookup API. |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/README.md |
Documents reading patterns and updates examples. |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/Microsoft.Extensions.Documents.Abstractions.json |
Records the experimental API changes. |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/ExtractionDocumentTable.cs |
Enumerates anchors only and exposes GetCoveringCell(). |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/ExtractionDocumentExtensions.cs |
Implements recursive text extraction with cycle checks. |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/ExtractionDocumentElement.cs |
Adds section traversal and clarifies image-text documentation. |
src/Libraries/Microsoft.Extensions.Documents.Abstractions/ExtractionDocument.cs |
Clarifies traversal limits and shares the section walker. |
Review effort: Balanced
adamsitnik
reviewed
Nov 20, 2026
adamsitnik
left a comment
Member
There was a problem hiding this comment.
@luisquintanilla PTAL at my comment, thanks!
Per review feedback and the Framework Design Guidelines, prefer instance methods over extension methods on types we own. GetText is now a non-virtual instance method on ExtractionDocument, ExtractionDocumentElement and ExtractionDocumentTableCell; the shared traversal lives in an internal helper and the public ExtractionDocumentExtensions class is removed. Call sites written as x.GetText() compile unchanged. Also make the README role-filter example compare roles case-insensitively, as the README recommends. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
GetText() stays non-virtual. Custom element types can optionally override GetAdditionalText() to contribute text stored outside their TextContent payload; the walker keeps ordering, separators and cycle detection. Image Text is now read through the same hook, so output is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
luisquintanilla
marked this pull request as ready for review
November 20, 2026 19:45
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #7962
Summary
Make reading an
ExtractionDocumentcomplete and independent of how a provider shapes its sections. Today the obvious loop silently drops text, and how much it drops depends on the provider: on the same 10 PDFs it reads 76–83% of the words from flat providers but only 5–6% from the Azure providers. This PR addsGetText(), addsEnumerateContent()to sections, makes table-cell enumeration anchor-only, and documents where text can live.The hierarchy is unchanged. This PR changes only
Microsoft.Extensions.Documents.Abstractions, and every new API is[Experimental]like the rest of the package.Why: two ways to lose text
The model from #7835 lets each producer choose the section shape. Some emit one flat section per page; others nest sections under headings, and table cells can themselves contain sections. Illustrative shapes:
The loop most people write first:
ExtractionDocumentTextElement. Text also lives in table cells (whose content can itself be a section), inExtractionDocumentImageElement.Text, and in the localContentof the document, sections and tables.Before this PR, no public API returned "all the text" and the docs did not say where text can live, so every consumer had to write a correct recursive walker by hand. Our own docs and samples did not.
Measured with five providers on 10 hard pages from ParseBench-2. The number is the share of each page's ground-truth words a strategy returns:
document.GetText()Flat providers don't change, so nothing regresses. Compare within a row; this isn't a provider ranking. The other strategies, per-page notes and how to reproduce are under Evidence below.
What this PR adds
ExtractionDocument.GetText()ExtractionDocumentElement.GetText()ExtractionDocumentTableCell.GetText()protected virtual string? ExtractionDocumentElement.GetAdditionalText()TextContentpayload, whichGetText()reads after the payload. The default returnsnull.ExtractionDocumentImageElementoverrides it to returnText.ExtractionDocumentSection.EnumerateContent()ExtractionDocument.EnumerateContent(), scoped to a section.ExtractionDocumentTableCellCollection.GetCoveringCell(int, int)ExtractionDocumentTableCellCollection : IEnumerable<ExtractionDocumentTableCell>IEnumerable<ExtractionDocumentTableCell?>. Enumeration yields anchor cells only, with nonullplaceholders for empty or covered positions. The indexer is unchanged.The three
GetText()methods are non-virtual instance methods, following the Framework Design Guidelines' preference for instance methods over extension methods on types you own (review feedback). They share one internal traversal, so every overload has the same semantics. Custom element types extend it through the protected virtualGetAdditionalText()hook rather than by overridingGetText(), so ordering, separators and cycle detection stay in one place. Overriding is optional: a custom element'sTextContentpayload is read without an override.Docs:
Sections,Elements,EnumerateContentandImageElement.Textnow say that producers choose the section shape and where text can live.GetText().Reading a document after this PR
document.GetText().foreach (var e in document.EnumerateContent())and calle.GetText()on every element; don't keep only text elements.Elements,table.Cells(anchors) andGetCoveringCell, as before.Note
The obvious two-level loop above still compiles and is still lossy for nested providers. The hierarchy stays on purpose: flattening it would lose information that chunkers, citation and layout consumers need. This PR makes the correct path the easy one rather than making the wrong one impossible. It mirrors LINQ to XML (
Elements()/Descendants()/Value). An analyzer that flags the loop is API question 4.How this fits with the other PRs
DocumentExtractionClient; it is unchanged.Compatibility
Cellselement type changes from nullable to non-nullable. Existingcell?.Contentandif (cell is null)checks still compile; they become dead code. Code that relied onCells.Count() == RowCount * ColumnCountbreaks; use the indexer orGetCoveringCell. The package is experimental and preview.foreach (var cell in table.Cells)no longer yieldsnullfor empty or covered positions.IngestionDocumentElementExtensions.EnumerateSourcesandImageAlternativeTextEnricher.EnumerateImages(Integrate shared Documents while retaining IngestionDocument #7904) already usecell?.Content, so they are unaffected.11.0.0-preview.1.26560.3.Suggested review focus
GetTextsemantics (ExtractionDocumentText.cs): what's included, order, separator. The full rules are underGetTextsemantics below.Cells(ExtractionDocumentTable.cs): the one breaking change, plusGetCoveringCellbecoming public.ExtractionDocument.cs,ExtractionDocumentElement.cs,README.md): do they make the shape-independent path obvious?DocumentTraversalTests.cs): especially the provider-shape test, which asserts that flat, nested and cell-nested shapes of the same content produce identicalGetText().Validation
DocumentTraversalTestsplus updated table tests: 239/239 pass on all TFMs.Microsoft.Extensions.Documents.Abstractions.json) regenerated.GetAdditionalTexthook) against the build before the hook, on the same 10 pages: all 174 per-document output files for PdfPig, Content Understanding and Document Intelligence are byte-identical, including 24 image elements whose text now goes through the hook. Mistral differs on 3 pages, all with no images and with different provider structure between calls (OCR run-to-run variation). In the final run, D == E on all 45 extracted documents across the 5 providers, and A == E for every flat provider.Questions for API review
EnumerateContent. "Content" collides with theContentpayload property, and the method doesn't return payloads.EnumerateLeavesorDescendants(withDescendantsalso yielding sections, as in LINQ to XML)?Environment.NewLinevs"\n"for deterministic output across operating systems (chunk hashes, snapshots).image.Text(OCR) and excludeAlternativeText(caption)? Or add an option or overload?foreach … in document.Sections→foreach … in section.Elementswith anis ExtractionDocumentTextElementfilter and no recursion, and offerGetText()/EnumerateContent()as the fix?Cellsenumeration break. Is anchor-only enumeration acceptable in preview, or should it be a newAnchorsproperty withCellsleft as is?ImageElement.Textvs.GetText(). The guidelines say not to have a property with the same name as aGetmethod, and image elements now have both, with different values:Textis the recognized text alone, whileGetText()returns localTextContentfollowed byText. Rename the property (for exampleRecognizedText), rename the method, or keep both and rely on the docs?GetText()is non-virtual; custom elements contribute text throughprotected virtual string? GetAdditionalText(), read after the payload and before children. Overriding is optional, and adding the hook is non-breaking. Is the name right (GetAdditionalText,GetOwnText,GetLocalText)? And should the hook also cover custom containers, whose children neitherGetText()norEnumerateContent()traverse today?Evidence: live measurements (5 providers × 10 ParseBench-2 pages)
Mean ground-truth token recall over 10 single-page PDFs from
llamaindex/ParseBench-2: charts, complex table headers ×2, dense, financial, form, headings, layout, timetable and CJK pages. Every provider was created through the public samples repo, and each result was read five ways:EnumerateContent+is TextElementEnumerateContent+GetText()document.GetText()GetText()0.99 for both Azure providers.Reproduce: the harness, pinned corpus and per-page numbers are in
validation/parsebench(luisquintanilla/dotnet-document-extraction-samples#3). Running it writes each document's structure and the text every strategy read.GetTextsemanticsTextContentcomes before its children.image.Textand any custom element'sGetAdditionalText(), after the element's payload.AlternativeText, bytes/references, metadata and raw representations.Environment.NewLine.InvalidOperationExceptionon cycles or null entries, matchingEnumerateContent.Design: why not make the loop impossible?
The hierarchy is unchanged on purpose: flattening it would lose information that chunkers, citation and layout consumers need. So this PR makes the correct path the easy one rather than making the wrong one impossible, in the same way LINQ to XML offers
Elements(),Descendants()andValueover one tree.Alternatives considered and rejected:
Subsectionscollection. Two collections to keep in sync, and the loop is still lossy.Elementswith parent pointers. A large redesign that conflicts with shared nodes, and it still doesn't fix type filtering.Review map (10 files, +648 / −71)
All paths are under
src/Libraries/Microsoft.Extensions.Documents.Abstractions/ortest/Libraries/Microsoft.Extensions.Documents.Abstractions.Tests/.ExtractionDocumentText.csGetText()methods.ExtractionDocumentElement.csExtractionDocumentElement.GetText(), the protected virtualGetAdditionalText()hook (overridden by the image element) andExtractionDocumentSection.EnumerateContent(); docs onElementsandImageElement.Text.ExtractionDocument.csGetText()with the full semantics docs; docs onSectionsandEnumerateContent(section shape, where text lives);EnumerateSectionbecomes internal for reuse.ExtractionDocumentTable.csGetCoveringCell; newExtractionDocumentTableCell.GetText().README.mdGetText(); notes the custom-element hook.Microsoft.Extensions.Documents.Abstractions.jsonDocumentTraversalTests.csEnumerateContent(nesting, independence, cycles/nulls); anchor-onlyCellsplusGetCoveringCellover every position;GetTextorder across every text location, empty blocks, shared content, cycles, null entries; custom elements with and without the hook; provider-shape equivalence.DocumentTableCellCollectionTests.cs,DocumentTableElementTests.csDocumentReferencesTests.csGetCoveringCellis public.Microsoft Reviewers: Open in CodeFlow