Skip to content

Reindex after a Searchable flip is synchronous and scales with the whole tenant #2

Open

Description

@duguankui

Status

A known trade-off, accepted for now (see the comment on ContentTypeManager.GetSearchableFieldIds). Revisit when one of the conditions under "Revisit when" is met; until then there is nothing to do.

What happens

Searchable is a per-usage flag: it lives on ContentTypeField, not on the Field definition, because the same field can be searchable in one content type and not in another (design doc §2.3). Contents saved before a flip hold no index rows for the field, so filtering on it would silently return an incomplete result set; and the rows left behind by a flip to off would keep matching. ContentTypeManager.UpdateAsync therefore re-derives the index when the set of searchable fields of the type changes (a change of Required, ShowInList, order or label does not).

It does so with IFlexFieldIndexManager<Content>.RebuildAsync(), the only bulk entry point the kernel offers:

  • it visits every content of the tenant, in pages of 100 ordered by id, flushing once per page. The kernel cannot push a predicate down, so contents the change cannot affect are visited too;
  • per content (EF Core provider) it re-resolves the content type and its field definitions, reads the content's index rows, deletes them and inserts the projection. It always reports a write, so every page flushes;
  • it runs inside the request that changed the content type (the admin UI or an MCP tool call), in that request's unit of work.

The result is correct and, under the default transactional unit of work, all or nothing. But the cost grows with all the contents of the tenant, not with the contents of the type that changed, and nothing bounds how long the request takes.

Why it is not worked on now

  • No measurement. There is no number for what a flip costs at any size, and no report of one being slow. Without one the cost cannot be weighed against the fix.
  • The failure mode of the current code is latency; the failure mode of an optimization is a silently incomplete index. The rebuild is deliberately obvious: re-derive everything. Anything that scopes it has to get the scope exactly right.
  • It is rare. Only a change of the set of searchable fields triggers it, which is schema work and tends to happen early in a site's life. (An AI client iterating on a schema over MCP may flip more often than a person would.)

Revisit when

Any one of:

  • a flip measured on a representative tenant takes a meaningful share of the request / MCP client timeout (set the number once it is measured);
  • a flip is reported slow, or a request times out;
  • (option 3 only) a second downstream of the kernel needs a scoped rebuild.

Measure first

Seed a tenant with N contents spread over a few content types (SQLite is enough to see the shape) and time a flip on the smallest type. The same measurement shows whether the cost is dominated by the scope (the rows of other types) or by the synchronous walk itself (one large type), which decides between the options below.

Options

  1. Scope it inside Site. Only the contents of the changed type can be affected, since Searchable is a flag of that type's own arrangement. In ContentTypeManager, page through those contents and call SynchronizeAsync on each. IContentRepository.GetListAsync(contentTypeId: ..., skipCount, maxResultCount) already filters and pages, and its default order (PublishTime desc, Id desc) is deterministic, so no new repository method is needed. Things to get right:

    • SynchronizeAsync deliberately does not flush (the derived state commits with the value bag), so the loop has to save per page itself, or the whole type's changes pile up until the request ends;
    • the loop repeats the kernel's own paging and termination (a short or empty page ends it);
    • ContentTypeManager needs IContentRepository (and the unit of work): a new constructor parameter is a public API change under SemVer, so use property injection or accept it;
    • it fixes the uneven case (a small type next to a large one), not a large type.

    A test can see this fail today: flip a flag on the gallery type and assert that the index row ids of the article type's contents are unchanged (CreateIndexRow generates a new id every time, so a full rebuild replaces them). Should_Reindex_Existing_Contents_When_A_Field_Becomes_Searchable keeps guarding the other half.

  2. Make it asynchronous. Run the walk as a background job, in batches, resumable. This is the fix for the unbounded request, which scoping is not; it also needs an answer for the window in which the index is incomplete.

  3. A scoped rebuild in the kernel (Dignite.Abp.FlexFields), for example an overload taking a sequence of entity ids. A predicate overload cannot work: the kernel documents that it has no way to push a bag predicate down through a downstream's own query. It means a new kernel API, a release, a MongoDB implementation and a version bump here, so it is only worth it if a second downstream wants it.

If a condition above is met: measure, then take option 1 if the cost is dominated by the scope, option 2 if it is dominated by one large type.

The other RebuildAsync caller, FieldManager.UpdateAsync on a field type change, has the same shape. The admin UI does not allow changing a field's type after creation (only the API or MCP can), and scoping it would need a field-to-content-types lookup that no longer exists, so it is out of scope here.

References

  • src/Dignite.Site.Domain/ContentTypes/ContentTypeManager.cs (UpdateAsync, GetSearchableFieldIds)
  • src/Dignite.Site.Domain/Contents/ContentFlexFieldProvider.cs
  • src/Dignite.Site.EntityFrameworkCore/Contents/EfCoreContentRepository.cs (GetPagedListOrderByIdAsync, GetListAsync)
  • Kernel (abp-modules/flex-fields): FlexFieldIndexManagerBase, FlexFieldEntityPager, EfCoreFlexFieldIndexManagerBase
  • Regression test: Should_Reindex_Existing_Contents_When_A_Field_Becomes_Searchable
  • Design doc §2.3, §2.4

Reframed on 2026-10-29. It was filed as "Narrow the Searchable-flip reindex to the affected content type", on the scope alone. The unbounded synchronous walk is the larger concern, and a measurement and a revisit condition were missing.

Activity

  1. added this to the P0 Content kernel milestone on Aug 4, 2026
  2. added
    phase: P0Content kernel - landed, these are follow-ups
    tech debtKnown shortcut taken during P0
    group: 0-kernel-debtIndependent P0 cleanup - blocks nothing, insert anywhere
    on Aug 4, 2026
  3. added
    group: 9-laterParked until the product is further along
    and removed
    group: 0-kernel-debtIndependent P0 cleanup - blocks nothing, insert anywhere
    on Oct 29, 2026
  4. changed the title [-]Narrow the Searchable-flip reindex to the affected content type[/-] [+]Reindex after a Searchable flip is synchronous and scales with the whole tenant[/+] on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: content-kernelPage / ContentType / Content / Fieldgroup: 9-laterParked until the product is further alongphase: P0Content kernel - landed, these are follow-upstech debtKnown shortcut taken during P0

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions