[v25.3.x] cluster/health: use kafka high watermark in partition_status - #30950
Merged
Conversation
build_partition_status() reported a partition's high watermark as
from_log_offset(p.high_watermark()), the raw local-log position. For a
read replica the local raft log holds only the synced manifest (no data
batches), so this translates to ~0 in the Kafka offset space instead of
the real cloud high watermark (next_cloud_offset).
The consumer-group lag metric reads this health-report HWM, so
redpanda_kafka_consumer_group_lag_{max,sum} collapsed to 0 for
read-replica topics regardless of the committed offset, hiding real
backlog. The Kafka-visible HWM (rpk / list_offsets) was unaffected
because it already goes through kafka_high_watermark().
Use cluster::kafka_high_watermark(p) so high_watermark is cloud-aware
and consistent with the log_start_offset set alongside it. For
non-read-replica partitions the value is unchanged.
Covered by a ducktape test
(ReadReplicaConsumerLagTest.test_group_lag_metrics_read_replica in
consumer_group_test.py): it pins the committed offset to 0 so the
reported lag reduces to the health-report HWM, then asserts that lag
equals the cloud-visible HWM. The read-replica setup helpers were
extracted from TestReadReplicaService into a ReadReplicaE2EBase base
class so the new test can reuse them without inheriting the e2e suite.
Follow-up to CORE-16393.
(cherry picked from commit f4352ad)
bartoszpiekny-redpanda
added
area/redpanda
kind/backport
andrwng
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #30873
Note: be aware that this fix won't be ported to v25.2.x as high_watermark on this branch is already cloud-aware.
Conflict details
rptest.services.redpandaimport block — the target branch already importsRedpandaService, while the cherry-picked commit addsget_cloud_storage_type. Resolved by keeping both imports.ReadReplicaE2EBasebase class out ofTestReadReplicaService, but the upstream diff also assumed read-replica test infrastructure that is not present on this branch (CrossRegionRRRTestMark/cross_region_rrr_test, theReadReplicaSourceModemodeparameter, andcloud_topics_long_term_flush_interval). Resolved by introducingReadReplicaE2EBaseadapted to the target branch's existing structure (without the cloud-topics / cross-region infra) and by adding areplication_factorparameter to_setup_read_replica, which the newReadReplicaConsumerLagTestrequires.