schema_registry: load _schemas once across shards on startup by sjust-redpanda · Pull Request #31137 · redpanda-data/redpanda

sjust-redpanda · 2026-07-16T20:55:21Z

Schema Registry recovery of the internal _schemas topic on startup could
run multiple redundant full replays concurrently on the reader shard, making
cold start of a large registry take much longer than necessary (recovery
time scaled with the number of racing shards).

service is a per-shard peering_sharded_service, so each shard owns its
own _ensure_started one_shot, and do_start() only stamps _is_started
(its re-entry guard) from inside its invoke_on_all, after
create_internal_topic(). Previously, when first requests landed on several
shards within that window, each shard ran its own do_start() and launched
an independent replay.

This PR instead funnels every shard through a single one_shot on the reader
shard so the topic is loaded exactly once; the per-shard one_shot still
caches completion, so only the first request on each shard pays a
cross-shard hop and steady state is unchanged.

Recovery is also lazy (nothing loads the topic until the first request), so
the first request after a restart can block for the whole replay. This PR
adds a cluster config, schema_registry_replay_on_startup (default off),
that proactively drives the single-flight load at startup, fire-and-forget
under the gate, so the store hydrates ahead of traffic without blocking
broker start-up.

Covered by ducktape tests: a single replay per restart under concurrent
multi-shard load, and the eager/lazy behavior of the new config.

Backports Required

v26.2.x
v26.1.x

Release Notes

Improvements

Schema Registry now replays the internal _schemas topic exactly once
when recovering on startup, instead of running redundant concurrent
replays; cold start of a large registry is significantly faster.
Added a schema_registry_replay_on_startup cluster property (default off)
that hydrates the Schema Registry store at broker start-up rather than
lazily on the first request.

WillemKauf · 2026-07-16T22:23:51Z

 }

 ss::future<> service::do_start() {
-    if (_is_started) {


is it worth keeping _is_started around? maybe as an extra vassert(!_is_started) sanity check?

I don't think so, but I don't feel strongly about it. Want me to re-add it?

There is a difference in behavior related to one_shot. If the fetch fails, the one_shot will fail all waiters (transitively to the per-shard one_shot) and then rearm. A subsequent attempt could then retry and actually succeed. Don't have tests for that case, and do_start already retries, so this may not be important.

ohhh sorry i see now the special one_shot typedef. I was missing that. SGTM

WillemKauf

LGTM, but i'd defer to @pgellert signing off on this too. Are there other SR experts we can start pulling into these reviews so we aren't overloading poor Gellert? 🙂

vbotbuildovich · 2026-07-17T00:31:29Z

CI test results

test results on build#87257

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	ShadowLinkUnfinalizedUpgradeTest	test_full_lifecycle	null	integration	https://buildkite.com/redpanda/redpanda/builds/87257#019f6d13-0d8b-478a-985f-773431ba6136	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0385, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1110, p1=0.3083, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkUnfinalizedUpgradeTest&test_method=test_full_lifecycle

test results on build#87352

test_status	test_class	test_method	test_arguments	test_kind	job_url	passed	reason	test_history
FLAKY(PASS)	ShadowLinkTopicFailoverTests	test_link_failover	{"source_cluster_spec": {"cluster_type": "kafka", "kafka_quorum": "COMBINED_KRAFT", "kafka_version": "3.8.0"}, "storage_mode": "tiered_v2", "with_failures": false}	integration	https://buildkite.com/redpanda/redpanda/builds/87352#019f7240-bd7b-4858-9d5f-37769de5697c	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkTopicFailoverTests&test_method=test_link_failover
FLAKY(PASS)	NodesDecommissioningTest	test_decommission_status	null	integration	https://buildkite.com/redpanda/redpanda/builds/87352#019f7240-bd7f-469d-a935-5da333243820	10/11	Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000)	https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodesDecommissioningTest&test_method=test_decommission_status

pgellert · 2026-07-17T12:30:46Z

+                    "eager _schemas replay failed at start-up; will retry on "
+                    "the first request: {}",


will retry on the first request

Is this bit true? The load once ensures that we will not retry reloading (which is fine), so I guess we may only "retry" in the sense that we will continue to try to replay the topic from the read syncs on the http handler path. Perhaps that's what you meant?

If the fetch fails, the one_shot will fail all waiters (transitively to the per-shard one_shot) and then rearm. A subsequent attempt triggered by a request could then retry and actually succeed. Don't have tests for that case, and do_start already retries, so this may not be important.

pgellert · 2026-07-17T14:31:15Z

Are there other SR experts we can start pulling into these reviews so we aren't overloading poor Gellert?

@nguyen-andrew has done a bunch of work on SR, so feel free to throw him on reviews, too. Or anyone from the enterprise team really (@redpanda-data/core-enterprise, we're about to get everyone into this GH team) can do at least a single pass and bubble specific questions up to me. :)

service is a per-shard peering_sharded_service, so each shard has its own _ensure_started one_shot. do_start() only stamps _is_started (its re-entry guard) from inside its invoke_on_all, after create_internal_topic(), so first requests arriving on several shards concurrently each run do_start() and launch an independent, redundant full replay of the _schemas topic on the reader shard - slowing recovery by up to the shard count. Funnel every shard through a single one_shot on the reader shard so the topic is loaded exactly once. The per-shard one_shot still caches completion, so only the first request on each shard pays a cross-shard hop; failure/retry semantics are unchanged. Covered by a ducktape test that restarts the registry under concurrent multi-shard load and asserts exactly one replay runs per restart. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

_load_once makes do_start() run exactly once on the reader shard, so its old cross-shard re-entry guard is redundant. Drop the _is_started flag, assert do_start() runs on the reader shard, and run the fetch inline rather than via invoke_on_all (whose only remaining job was setting _is_started on every shard). Removing _is_started also fixes a latent case: a fetch that exhausted its retries left _is_started set, so the _load_once retry short-circuited and the store never loaded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

nguyen-andrew

🙌

nguyen-andrew · 2026-07-17T20:27:43Z

+      *this,
+      "schema_registry_replay_on_startup",
+      "Replay the internal `_schemas` topic into the Schema Registry store at "
+      "broker start-up instead of lazily on the first request. Makes recovery "


nit: I think instead of "at broker start-up", we could say "at Schema Registry start-up", since I believe we can restart the schema registry service (and have this new config take effect) without having to restart the broker

Good point, fixed.

Recovery of the _schemas store is lazy: nothing loads the topic until the first request (or an internal schema::registry access) reaches ensure_started(). On a large registry the first request then blocks for the whole replay, and there is no way to hydrate the store ahead of traffic. Add a cluster config (default off) that, when set, proactively drives the replay on the reader shard at service start-up, fire-and-forget under the gate so it does not block broker start-up. It reuses the single-flight ensure_started() path, so a request that races the eager trigger still results in exactly one replay. Covered by ducktape tests for both settings: with the config set the store hydrates on restart with no client request, and with it unset recovery stays lazy until the first request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

vbotbuildovich · 2026-07-18T04:39:10Z

/backport v26.2.x

vbotbuildovich · 2026-07-18T04:39:11Z

/backport v26.1.x

vbotbuildovich · 2026-07-18T04:39:34Z

Branch name "v26.2.x" not found.

Workflow run logs.

sjust-redpanda · 2026-07-24T00:29:22Z

/backport v26.2.x

sjust-redpanda requested a review from a team as a code owner July 16, 2026 20:55

github-actions Bot added area/build area/redpanda labels Jul 16, 2026

sjust-redpanda force-pushed the sjust/sr-startup-race branch from 4653902 to 0e4165b Compare July 16, 2026 21:16

sjust-redpanda marked this pull request as draft July 16, 2026 21:44

sjust-redpanda force-pushed the sjust/sr-startup-race branch from 0e4165b to 714c62d Compare July 16, 2026 21:52

sjust-redpanda marked this pull request as ready for review July 16, 2026 22:15

sjust-redpanda requested review from WillemKauf and pgellert July 16, 2026 22:15

WillemKauf reviewed Jul 16, 2026

View reviewed changes

WillemKauf previously approved these changes Jul 17, 2026

View reviewed changes

pgellert previously approved these changes Jul 17, 2026

View reviewed changes

sjust-redpanda and others added 2 commits July 17, 2026 11:42

sjust-redpanda dismissed stale reviews from pgellert and WillemKauf via a553561 July 17, 2026 18:54

sjust-redpanda force-pushed the sjust/sr-startup-race branch from 714c62d to a553561 Compare July 17, 2026 18:54

nguyen-andrew previously approved these changes Jul 17, 2026

View reviewed changes

sjust-redpanda dismissed nguyen-andrew’s stale review via e9aae73 July 17, 2026 22:30

sjust-redpanda force-pushed the sjust/sr-startup-race branch from a553561 to e9aae73 Compare July 17, 2026 22:30

sjust-redpanda requested review from nguyen-andrew and pgellert July 17, 2026 23:16

nguyen-andrew approved these changes Jul 18, 2026

View reviewed changes

sjust-redpanda merged commit 0dc8147 into redpanda-data:dev Jul 18, 2026
19 checks passed

vbotbuildovich mentioned this pull request Jul 18, 2026

[v26.1.x] schema_registry: load _schemas once across shards on startup #31174

Merged

dotnwat reviewed Jul 20, 2026

View reviewed changes

Comment thread src/v/pandaproxy/schema_registry/service.cc Outdated

vbotbuildovich mentioned this pull request Jul 24, 2026

[v26.2.x] schema_registry: load _schemas once across shards on startup #31266

Closed

tyson-redpanda mentioned this pull request Jul 24, 2026

[v26.2.x] Cherry-pick dev PRs since branch point #31294

Merged

1 task

		"eager _schemas replay failed at start-up; will retry on "
		"the first request: {}",

Uh oh!

Conversation

sjust-redpanda commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backports Required

Release Notes

Improvements

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillemKauf left a comment

Choose a reason for hiding this comment

Uh oh!

vbotbuildovich commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI test results

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgellert commented Jul 17, 2026

Uh oh!

nguyen-andrew left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vbotbuildovich commented Jul 18, 2026

Uh oh!

vbotbuildovich commented Jul 18, 2026

Uh oh!

vbotbuildovich commented Jul 18, 2026

Uh oh!

Uh oh!

sjust-redpanda commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sjust-redpanda commented Jul 16, 2026 •

edited

Loading

vbotbuildovich commented Jul 17, 2026 •

edited

Loading