Skip to content

[SPARK-59747][CORE][4.1] Don't retry compaction of rolled event log files with malformed input - #59619

Closed
DengYiping wants to merge 1 commit into
apache:branch-4.1from
DengYiping:ydeng/spark-59747-branch-4.1
Closed

DengYiping wants to merge 1 commit into
apache:branch-4.1from
DengYiping:ydeng/spark-59747-branch-4.1

Conversation

@DengYiping

@DengYiping DengYiping commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Backport #58996 to branch-4.1.

In FsHistoryProvider.compact(), catch CharacterCodingException thrown while compacting rolled event log files, log it at WARN, and still record lastEvaluatedForCompaction for the current index.

EventLogFileCompactor.initializeBuilders also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

The conflict resolution retains this branch's existing single fileCompactor.

Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, EventLogFileCompactor fails with a MalformedInputException while replaying them to initialize the event filter builders. compact() only records lastEvaluatedForCompaction after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

Does this PR introduce any user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

How was this patch tested?

Added a new test to FsHistoryProviderSuite. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, lastEvaluatedForCompaction is recorded, and a WARN names the malformed file.

Passed locally with Java 21 on the branch-4.1 backport:

build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \
  'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \
  'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite'

Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; git diff --check also passed. LevelDB-backed variants were not run on Apple Silicon.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport.

@DengYiping DengYiping changed the title [SPARK-59747][CORE][4.1] Don’t retry compaction of rolled event log files with malformed input [SPARK-59747][CORE][4.1] Don't retry compaction of rolled event log files with malformed input Oct 20, 2026
@DengYiping
DengYiping marked this pull request as ready for review October 20, 2026 00:30
@sarutak

sarutak commented Oct 20, 2026

Copy link
Copy Markdown
Member

Thank you for opening this backport-PR, @DengYiping.
Since this is a backport-PR for the same issue as #58996, please make the description as similar to #58996's as possible, and say "Backport #58996 to branch-4.1" at the top of the What changes were proposed in this pull request? section (no need to mention me in the description).

@DengYiping

Copy link
Copy Markdown
Contributor Author

Thanks! Both backport descriptions now follow #58996, with the backport line at the top.

@DengYiping

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @sarutak. Build/test jobs passed, but the test reporter still fails. Is that acceptable for this backport, or should it be fixed first?

@sarutak

sarutak commented Oct 27, 2026

Copy link
Copy Markdown
Member

Hi @DengYiping, I think the CI failure is not relevant to this change but could you rebase your branch to branch-4.1 and push it again? Thanks.

@DengYiping
DengYiping force-pushed the ydeng/spark-59747-branch-4.1 branch from d9c7518 to 2a8e0f0 Compare October 27, 2026 21:12
@sarutak

sarutak commented Oct 28, 2026

Copy link
Copy Markdown
Member

I believe the CI failure is not relevant this change and am going to merge this.

sarutak pushed a commit that referenced this pull request Oct 28, 2026
…iles with malformed input

### What changes were proposed in this pull request?

Backport #58996 to `branch-4.1`.

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

The conflict resolution retains this branch's existing single `fileCompactor`.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file.

Passed locally with Java 21 on the `branch-4.1` backport:

```sh
build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \
  'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \
  'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite'
```

Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport.

Closes #59619 from DengYiping/ydeng/spark-59747-branch-4.1.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
@sarutak sarutak closed this Oct 28, 2026
@sarutak

sarutak commented Oct 28, 2026

Copy link
Copy Markdown
Member

Merge Summary:

Posted by merge_spark_pr.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants