[v25.3.x] admin/controller: expose raft0 cancel_reconfiguration - #31198
Merged
Conversation
Exploses an api which allows the caller to cancel the current raft0 reconfiguration. This operation preserves any additional enqueued reconfigurations. This is to be used as an emergency escape hatch if a raft0 reconfiguration is stuck such as the case of a dead learner. (cherry picked from commit 9570661)
... via evil_mode Adds an evil_mode parameter which allows partition_force_reconfiguration to operate against the controller. It replicates a fresh raft0 configuration whose revision is past the current log tail, superseding every change currently enqueued in the members_manager raft0 queue (the members backend drops them as stale). If the new configuration removes the processing leader from the voter set, it steps down so a survivor can take over. This is a last ditch fix for controller raft issues (cherry picked from commit aa2f903)
Adds ducktape coverage for the controller (raft0) reconfiguration admin APIs exposed in this series: - cancel_reconfiguration unwedges a stuck raft0 learner and drains the adds queued behind it, without stranding queued membership changes. - successive cancels clear a whole backlog of dead learner adds. - evil_mode force reconfiguration nukes the enqueued raft0 changes and removes stuck learners or a healthy voter; forcing away from the current leader steps it down and transfers leadership to a survivor. - a force-removed voter is then cleaned out of the members table by a follow-up decommission. (cherry picked from commit b2c3ba8)
If you're holding the lock, you should probably also hold the gate. Both of these are top level lock requiring apis that were not taking the gate leaving them vulnerable to shutdown races. Solution: take the gate (cherry picked from commit e44a8c2)
refresh_commit_index grabs the lock and yields while not holding a gate holder. consensus can be torn down underneath if you're holding a lock you should probably hold a gate (cherry picked from commit dd3a5b8)
Collaborator
CI test resultstest results on build#87486 |
Contributor
There was a problem hiding this comment.
Pull request overview
Backports and extends admin/raft changes to provide operational “escape hatches” for a wedged controller (raft0), including admin-triggered cancel/force reconfiguration, plus consensus lifecycle (gate) correctness fixes to prevent teardown during async operations.
Changes:
- Extend admin partition endpoints to allow canceling raft0 reconfiguration and (with
evil_mode=true) forcing raft0 replica set replacement. - Add consensus gate-holding around cancellation/refresh paths and introduce a replicated force-replace configuration API.
- Expand raft0-throttling rptest coverage with new scenarios for cancel/force behavior and queued-add semantics.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/rptest/tests/throttled_raft0_test.py | Refactors/expands raft0 wedge tests; adds new cancel/force scenarios and helper methods. |
| tests/rptest/services/admin.py | Adds typed evil_mode support to force_set_partition_replicas via query params. |
| src/v/redpanda/admin/server.cc | Maps raft::errc::invalid_configuration_update to HTTP 400. |
| src/v/redpanda/admin/partition.cc | Enables controller (raft0) cancel and gated “evil mode” force reconfiguration paths. |
| src/v/raft/consensus.h | Declares replicated force-replace configuration and op-lock acquisition helper using std::expected. |
| src/v/raft/consensus.cc | Implements replicated force-replace; holds gate across cancel/refresh to avoid teardown races. |
| src/v/cluster/controller.h | Exposes controller APIs for canceling and forcing raft0 reconfiguration. |
| src/v/cluster/controller.cc | Implements raft0 cancel/force operations, including revision bumping for force-replace. |
Comment on lines
+174
to
+176
| def _joiner_in_brokers(self) -> bool: | ||
| """True if the joiner appears in any started node's broker list""" | ||
| return self._node_in_brokers(self.JOINER_NODE_ID) |
Comment on lines
+384
to
+386
| self.redpanda._admin.patch_cluster_config( | ||
| upsert={"raft_learner_recovery_rate": 100 * 1024 * 1024} | ||
| ) |
Comment on lines
+466
to
+489
| # Cancel the in-flight controller reconfiguration via the admin API. | ||
| # The request is routed to the raft0 leader rather than rejected. | ||
| def cancel_controller_reconfiguration() -> bool: | ||
| controller = self.redpanda.controller() | ||
| if controller is None: | ||
| return False | ||
| try: | ||
| self.redpanda._admin.cancel_partition_move( | ||
| namespace="redpanda", | ||
| topic="controller", | ||
| partition=0, | ||
| node=controller, | ||
| ) | ||
| return True | ||
| except Exception as e: | ||
| self.logger.debug(f"cancel reconfiguration not yet accepted: {e}") | ||
| return False | ||
|
|
||
| wait_until( | ||
| cancel_controller_reconfiguration, | ||
| timeout_sec=MEDIUM_TIMEOUT.timeout_s, | ||
| backoff_sec=MEDIUM_TIMEOUT.backoff_s, | ||
| err_msg="controller reconfiguration cancel was never accepted", | ||
| ) |
Comment on lines
+488
to
+510
| if (ntp == model::controller_ntp) { | ||
| // Reconfiguring the controller group is dangerous and gated behind an | ||
| // explicit evil_mode flag. It forcibly replaces raft0's configuration, | ||
| // blowing away any in-flight or enqueued raft0 reconfiguration. | ||
| if (!admin::get_boolean_query_param(*req, "evil_mode")) { | ||
| throw ss::httpd::bad_request_exception( | ||
| "Refusing to reconfigure the controller; pass evil_mode=true to " | ||
| "force a raft0 reconfiguration"); | ||
| } | ||
| std::vector<model::node_id> nodes; | ||
| nodes.reserve(replicas.size()); | ||
| for (const auto& bs : replicas) { | ||
| nodes.push_back(bs.node_id); | ||
| } | ||
| vlog( | ||
| adminlog.warn, | ||
| "evil_mode: forcing controller (raft0) reconfiguration to {}", | ||
| nodes); | ||
| auto err = co_await _controller->force_raft0_reconfiguration( | ||
| std::move(nodes)); | ||
| co_await throw_on_error(*req, err, model::controller_ntp); | ||
| co_return ss::json::json_void(); | ||
| } |
bharathv
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #30918
Also includes both raft gate-correctness fixes from #31022 (
raft: hold gate across cancel/force,raft: hold gate in refresh_commit_index) — same root issue; the v26.1.x backport (#30945) carried the first of these as well.Release Notes
Improvements
Backport adaptations:
tests/rptest/tests/throttled_raft0_test.py:RedpandaService.remove_from_started_nodeshas noreasonparameter on this branch — dropped the reason arguments (kept as comments).tests/rptest/services/admin.py: dev'sMaybeNodealias doesn't exist here — usedClusterNode | None(this branch's existing convention) in theforce_set_partition_replicassignature; rest of the typed signature kept verbatim.Verified locally: bazel build of
//src/v/redpanda/admin:admin,//src/v/raft,//src/v/clustersucceeds on this branch; branch-pinnedbazel run //tools:clang_formatproduces no changes;ruff format --check(CI-pinned 0.12.10) passes on the touched python files.