[emelialei88]

🎯 The "Why" (Intent)

Every block processor (writer thread) blocks until all replicants ack its commit, so the 8 writer threads cap commits in flight at 8. This moves that wait to a background thread; clients still get their reply only after every ack. Port of #1963.

🛠️ The "What" (Critical Changes)

• New seqnum-wait thread (async_dist_commit, on by default) checks acks for every handed-off commit, with the inline wait's timeouts, incoherent marking, lease and durable-LSN rules (shared code, split out of the inline wait).
• The block processor hands over the whole request (ireq and its request logger) and takes the next one; the reply, request log and osql cleanup still run after the acks. A request can now be finished from another thread.
• The waiter sends each reply itself. Commit replies now go out with net_send_nodrop, so a reply never waits for buffer space and one slow node cannot stall the waiter (Results 4).
• Queue cap async_dist_commit_max_outstanding_trans, default 1024 (Results 5). A commit that doesn't fit waits inline.
• Handed off: plain REP_SYNC_FULL blkseq commits, including durable LSNs and replicant_retry_on_not_durable. Everything else waits inline as before (Caveats).
• Early stop: an ack covers every earlier LSN, so within a pass the waiter skips commits waiting on a node an earlier commit is already waiting on (unless its deadline is due). Commits waiting on other nodes are still checked. Without it the waiter fell behind (older prototype, 64 MB, 128 clients: −12% vs off; with it 0%, and +15% at 512):

OLD prototype: skip everything after the first commit still waiting
  30  check → still waiting on B   → skip the rest of the pass
  35  skipped ✗                     A already acked it; reply waits ≤ 1.5 s

NEW: skip only commits waiting on a node already found behind
  30  check → still waiting on B   → behind = { B }
  35  waiting on A — A not in {B}  → check → acked → reply ✓ (same pass)
  45  waiting on B — B in {B}      → skip (B hasn't reached 30, so not 45)

• Durability counts only sanctioned nodes, as on main: the waiter gets the sanctioned list when it starts waiting and counts their acks separately, the same way the inline wait does (shared finish step).
• Hand-off metrics; async_dist_commit test comparing each case with the inline wait.
• async_dist_commit_bench test (long-running, so skipped by make) reruns every experiment below on any cluster and prints the numbers; debug_ack_delay_ms (test only, default 0) makes replicants send acks late for the late-ack runs.

async_dist_commit off (today)
  block processor ×8: [apply + commit] ─▶ [wait for all acks ......] ─▶ reply ─▶ next request

async_dist_commit on
  block processor ×8: [apply + commit] ─▶ hand off ─▶ next request
                                             │  (queue full / 2PC / not plain REP_SYNC_FULL ─▶ wait inline)
                                             ▼
  seqnum-wait thread ×1: queue (≤ cap) ─▶ check each node's ack ─┬─ too slow ─▶ mark incoherent
                                                                  └─ all acked ─▶ reply (nodrop) ─▶ client

📊 Results

8-node cluster, RelWithDebInfo build of this PR on current main, measured 2026-10-23 to 10-30. Each client inserts single rows (autocommit) into its own table, one commit in flight per client. 128 MB cache (production size; code default 64 MB), 8 writer threads, 48 SQL threads per node; 300 s × 2–3 unless noted.

1. Off vs on

xychart-beta
    title "Inserts/s: waiter on (bars) vs off (line)"
    x-axis "clients" [16, 32, 64, 128, 256]
    y-axis "inserts / s" 0 --> 24000
    bar [19701, 22043, 21531, 21890, 21858]
    line [14742, 14751, 14187, 15256, 14407]

Loading

numbers (+34% … +52%)

clients	off	on	gain
16	14,293 / 14,354 / 15,580	19,728 / 19,654 / 19,720	+34%
32	15,348 / 15,313 / 13,591	21,983 / 22,144 / 22,002	+49%
64	14,006 / 14,465 / 14,089	21,681 / 21,411 / 21,500	+52%
128	15,761 / 14,895 / 15,113	21,910 / 21,860 / 21,901	+43%
256	14,670 / 13,762 / 14,788	21,941 / 21,291 / 22,341	+52%
1000 (141 SQL threads)	13,647 / 14,720 / 13,666	19,115 / 19,462 / 19,261	+38%

Off is flat: 8 writer threads, each held for a whole ack round trip. The previous revision of this PR measured +16–25%; side by side on the same day (2 × 300 s) off is unchanged (32 / 256 clients: 14,361 / 13,358 before vs 14,106 / 14,467 now) and on went from 16,986 / 17,207 to 22,114 / 22,065.

2. Why not more writer threads, and #6293

• Before Take the trn repo lock only for logical commits #6293 (32 clients, 64 MB): 32 writer threads instead of 8: 12,276 → 4,471 (−64%), master CPU 6.5 → 14.9 cores; every ack wakes every sleeping thread. With the waiter but the old lock, 32 threads was still −21% vs 8 (11,003 vs 13,884). With the waiter, writers spent 50% of their time on the trn repo lock, which Take the trn repo lock only for logical commits #6293 now takes only for logical commits. Waiter alone +12%, Take the trn repo lock only for logical commits #6293 alone +16%, both +42% (cap 128).
• Now (waiter on, 2 × 180 s), 8 / 16 / 32 writer threads measure the same: 64 clients 21,456 / 21,149 / 21,408; 1000 clients 18,922 / 18,987 / 18,589.

3. Network latency: big gain. Slow replicants: none.

• Late acks (benchmark-only patch sends every replicant's ack D ms late; 64 clients, 2 × 120 s): off is capped at 8 threads ÷ D, on grows with clients. 1 ms 5,903 → 22,454 (3.8×); 2 ms 3,419 → 20,574 (6.0×); 5 ms 1,490 → 10,938 (7.3×).

xychart-beta
    title "Late acks, 64 clients: waiter on (bars) vs off (line)"
    x-axis "every ack delayed by" ["1 ms", "2 ms", "5 ms"]
    y-axis "inserts / s" 0 --> 24000
    bar [22454, 20574, 10938]
    line [5903, 3419, 1490]

Loading

• Slow replicants (every replicant sleeps 1 ms per message it receives; 16 clients, 2 × 120 s): 186 → 186. Throughput is capped by the replicants' apply rate; the waiter changes who waits, not how long. One slow replicant is marked incoherent and dropped from the wait, as today.

4. Who sends the reply

• The waiter thread sends every reply. A reply is ~100 bytes and at most 48 are pending per node (one per SQL thread), so with net_send_nodrop it is always queued at once, even when the 64 MB send queue to that node is full. That limit is meant for large messages (log records, schema), not replies.
• Before this, a reply to a node with a full queue was retried every 100 ms for up to 5 minutes; on the waiter thread that would have stalled every other commit. An earlier revision avoided it with a pool of 64 reply threads (healthy cluster, 256 clients: 17,103 vs 17,052 with the waiter replying, i.e. no speed difference); nodrop removes the blocking send instead, so the pool and its tunable are gone.
• Without the waiter, block processors reply as before, now also with nodrop.

5. Cap (only the cap changes; inline waits per run in brackets)

xychart-beta
    title "Cap sweep: 1000 clients (bars), 500 clients (line)"
    x-axis "cap" [128, 256, 512, 1024, 2048]
    y-axis "inserts / s" 0 --> 24000
    bar [17028, 17409, 19281, 19135, 19162]
    line [18857, 19230, 20706, 21088, 20902]

Loading

numbers (inline waits per run in brackets)

clients	128	256	512	1024	2048
500	18,857 (~72k)	19,230 (~32k)	20,706	21,088	20,902
1000	17,028 (~85k)	17,409 (~40k)	19,281	19,135	19,162

• A cap below the commits in flight costs ~10%: the overflow waits inline.
• 48 SQL threads × 8 nodes = at most ~384 commits in flight (8 before this change), so 512–2048 never fill. With 141 SQL threads per node (all 1000 clients in flight): cap 512 17,563 (−9%, ~19k inline waits), 1024 19,422, 2048 19,212.
• An unused cap costs nothing: a counter checked at hand-off; ~1 KB per queued commit.
• Master killed mid-load. 3 minutes into a 6-minute run the master is killed with kill -9; clients talk only to replicants and keep inserting. Every client has a commit in flight (cap = clients; SQL threads raised to 80 per node at 500 clients and 141 at 1000), 3 runs each:

clients	waiter	inserts/s before	inserts/s after	new master after	full speed again after	retried requests	failed requests	acked inserts lost
500	off	14.4–14.6k	15.8–16.0k	2.0–9.4 s	3 s	0–6	0	0
500	on	18.8–19.7k	20.7–21.5k	3.5–6.4 s	4–6 s	403–489	0–427	0
1000	off	11.3–12.7k	14.0–14.5k	8.2–8.6 s	5–8 s	4–7	0	0
1000	on	15.9–18.4k	18.7–19.7k	2.9–6.6 s	8 s	690–971	0–7,291	0

• After: 30–150 s after the kill. The new master keeps the gain. Full speed again: from the moment inserts stop until the first second back at 90% of the rate before the kill.
• Retried requests: commits the old master had written but not yet answered. The client resends them and the new master returns the saved result instead of inserting again. So it is about the number of commits waiting for acks at the kill: at most 8 without the waiter, up to one per client with it.
• Failed requests: cdb2api gave up after ~20 resends (rc -105); none of them were written. This is an artifact of the benchmark pinning each client to one replicant: they come in groups of ~143 (every client of one replicant), mostly before the kill, while the master had marked that replicant incoherent with the slow-replicant check, and pinned clients cannot move to another node. With unpinned clients (cdb2api picks the node, as real clients do; 1000 clients, 3 runs on / 2 off) no request failed: 16.4–18.1k before vs 11.8–12.3k off, 19.5–20.8k after vs 14.9–15.2k off, 570–748 retried requests vs 3–4.

6. Safety and cost

• durable_lsns on (now handed off): 32 clients 12,832 / 14,746 off vs 20,916 / 22,343 on (+57%); 128 clients 13,372 / 14,474 vs 22,173 / 22,026 (+59%). The durable / non-durable-retry / master-swing tests pass with the waiter turned on (final_non_durable_retry, lostwrite, master_load_commit, repeated_downgrades, rtcpu_drtest, cinsert_linearizable); @markhannum also ran Jepsen and the replicant-retry tests with the hand-off on under DURABLE_LSNS and replicant_retry_on_not_durable.
• Master killed mid-load: earlier, acked inserts were lost only when a second election followed the first, waiter on or off alike; fixed in Stop an election retry once a master is accepted #6359. The 12 kill runs above lost none.
• Master CPU: ~9 cores off vs 10–14 on, for 34–52% more commits.

7. A replicant falling behind (one replicant stopped, down 60 s, restarted; 1000 unpinned clients, cap 1000, 141 SQL threads, 900 s runs)

setting	behind at restart	coherent again after
waiter off (~14k inserts/s)	1260 MB	74 s
waiter on (~19–20k inserts/s)	1575 / 1533 / 1639 MB	472 s / 427 s / still 549 MB behind after 600 s
waiter on + logput_window 40000000	1615 / 1558 MB	82 s / 75 s
waiter on + logput_window + inmem_repdb	1776 / 1822 MB	104 s / 82 s

A restarted replicant gets every new log record twice: once as a live broadcast, which is gigabytes ahead of where it is, so it is parked as out of order on its single receive thread; and again from its own gap fill. Parking took 56% of that thread during catch-up, so at the waiter's commit rate it fell behind. logput_window (#6431) stops the live broadcast to an incoherent node more than 40 MB behind, so it only pulls the log in order. All settings tried and why are in docs/adr/0002-replica-catch-up.md in #6431.

⚠️ Caveats

• Inline, as before: quorum (sync atleast N), other sync modes, 2PC, rowlocks, and any request with DDL (its schema changes commit from their own threads on the same request, and only one thread may hand off).

• No gain when the replicants themselves are slow (section 3). The gain is from late acks over the network.

• A restarted replicant catches up slowly at full load with the waiter on unless logput_window is set: 427 s to never, vs ~80 s with it (section 7, Turn on logput_window by default #6431).

• On by default assumes logput_window (Turn on logput_window by default #6431) and inmem_repdb (Make inmem_repdb the default #6287) are on by default too. With the waiter alone, ~1000 commits in flight push replicants hard: the slow-replicant check (a node whose average ack time is 2× the second-worst) marked a node incoherent 23 and 15 times in 6-minute runs, and clients of that node are rejected until it catches up. With both on it did so 1 and 0 times, as with the waiter off (8 nodes, 1000 clients, 2026-10-30, runs interleaved):

  steady 6 min	inserts/s	slow-replicant marks
  waiter off	13,852 / 13,935	0 / 0
  waiter on	19,310 / 19,709	23 / 15
  waiter on + logput_window + inmem_repdb	23,134 / 22,769	1 / 0

  Master killed 3 minutes into a 6-minute run with all three on (2 runs): new master after 9.4 / 8.1 s, 19.2k / 18.8k inserts/s before and 21.2k / 21.9k after, 295 / 575 retried requests, 0 failed, 0 acked inserts lost; 5 / 3 slow-replicant marks per run, against 17–25 with the waiter alone.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
osql_cleanup [failed with core dumped]
analyze_partial_index_off_generated [failed with core dumped] **quarantined**
analyze [failed with core dumped] **quarantined**
tsa
consumer_non_atomic_default_consumer_generated **quarantined**
tunables
sc_downgrade [timeout] **quarantined**
skipscan [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: harness did not run - no test results were produced ⚠.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
comdb2sys_queueodh_generated [db unavailable at finish]
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
incremental_backup_usenames
reco-ddlk-sql **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
tunables

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
constraints_partial_index_off_generated [failed with core dumped]
constraints [failed with core dumped]
elect_retry_known_master
sqlqueries
skipscan [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
constraints_noreorder_generated [failed with core dumped]
sqlqueries
tunables
skipscan [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: 2/751 tests failed ⚠.

The first 10 failing tests are:
reco-ddlk-sql **quarantined**
nowritetimeout

[markhannum]

We've tested this extensively with the hand-off also enabled under DURABLE_LSNS and replicant_retry_on_not_durable (Jepsen plus the replicant-retry tests, on 6 nodes), and it works: the waiter already holds the reply until durability is known and returns ERR_NOT_DURABLE correctly. I think the !gbl_replicant_retry_on_not_durable and !BDB_ATTR_DURABLE_LSNS checks in the trans_commit_int() gate can be dropped.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

[roborivers]

Cbuild submission: Success ✓.
RoboMark testing: failed.
RoboRivers testing: Success ✓.

[roborivers]

Cbuild submission: Success ✓.
RoboMark testing: failed.
RoboRivers testing: Success ✓.

The first 10 failing tests are:
autoanalyze **quarantined**

[roborivers]

Failed to merge main into this PR - resolve conflicts and re-run ⚠.