Skip to content

[runtime] Fix prune state corner cases around checkpoints - #603

Merged
xintongsong merged 1 commit into
apache:mainfrom
joeyutong:codex/last-completed-seq-state
May 7, 2026
Merged

xintongsong merged 1 commit into
apache:mainfrom
joeyutong:codex/last-completed-seq-state

Conversation

@joeyutong

@joeyutong joeyutong commented Apr 2, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose of change

ActionExecutionOperator had two prune-state boundary issues around checkpoints.

  • prune bookkeeping used messageSequenceNumber, which tracks the latest started sequence, not the latest completed sequence. That means an in-flight sequence could be treated as prune-safe.

  • the operator called pruneState as soon as a run completed, before that completion was covered by a completed checkpoint.

Both issues are operator-side prune-boundary problems. In the current Kafka backend, pruneState only evicts the in-memory cache, so the correctness impact is mostly masked by replay/rebuild from the durable log. Still, the operator should not rely on that backend detail: the state-store prune boundary should only include completed sequences that are part of a completed checkpoint.

Fix

This change makes pruning checkpoint-safe again.

  • track a separate lastCompletedSequenceNumber keyed state
  • snapshot prune metadata from lastCompletedSequenceNumber instead of messageSequenceNumber
  • remove the eager prune path on run completion
  • prune action state only after notifyCheckpointComplete

Tests

The updated tests cover:

  • no pruning before checkpoint completion
  • no pruning of sequences that are still in flight
  • replay from an earlier checkpoint keeps durable state available
  • action state cleanup happens after checkpoint completion

API

No

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue. labels Apr 2, 2026
@joeyutong
joeyutong force-pushed the codex/last-completed-seq-state branch 3 times, most recently from 06356b2 to de49487 Compare May 7, 2026 09:46
@joeyutong
joeyutong force-pushed the codex/last-completed-seq-state branch from de49487 to 8f820d2 Compare May 7, 2026 10:59

@xintongsong xintongsong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xintongsong
xintongsong merged commit 74b5e08 into apache:main May 7, 2026
23 checks passed
joeyutong added a commit to joeyutong/flink-agents that referenced this pull request May 8, 2026
@joeyutong
joeyutong deleted the codex/last-completed-seq-state branch October 30, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants