Repository navigation
[SPARK-59747][CORE][4.1] Don't retry compaction of rolled event log files with malformed input - #59619
[SPARK-59747][CORE][4.1] Don't retry compaction of rolled event log files with malformed input#59619DengYiping wants to merge 1 commit into
Conversation
|
Thank you for opening this backport-PR, @DengYiping. |
|
Thanks! Both backport descriptions now follow #58996, with the backport line at the top. |
|
Thanks for the review, @sarutak. Build/test jobs passed, but the test reporter still fails. Is that acceptable for this backport, or should it be fixed first? |
|
Hi @DengYiping, I think the CI failure is not relevant to this change but could you rebase your branch to |
d9c7518 to
2a8e0f0
Compare
|
I believe the CI failure is not relevant this change and am going to merge this. |
…iles with malformed input ### What changes were proposed in this pull request? Backport #58996 to `branch-4.1`. In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. The conflict resolution retains this branch's existing single `fileCompactor`. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. Passed locally with Java 21 on the `branch-4.1` backport: ```sh build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \ 'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \ 'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite' ``` Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport. Closes #59619 from DengYiping/ydeng/spark-59747-branch-4.1. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org>
|
Merge Summary:
Posted by |
What changes were proposed in this pull request?
Backport #58996 to
branch-4.1.In
FsHistoryProvider.compact(), catchCharacterCodingExceptionthrown while compacting rolled event log files, log it at WARN, and still recordlastEvaluatedForCompactionfor the current index.EventLogFileCompactor.initializeBuildersalso logs a WARN naming the malformed event log file before rethrowing the decoding exception.The conflict resolution retains this branch's existing single
fileCompactor.Why are the changes needed?
When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8,
EventLogFileCompactorfails with aMalformedInputExceptionwhile replaying them to initialize the event filter builders.compact()only recordslastEvaluatedForCompactionafter compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.
With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.
Does this PR introduce any user-facing change?
Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.
How was this patch tested?
Added a new test to
FsHistoryProviderSuite. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created,lastEvaluatedForCompactionis recorded, and a WARN names the malformed file.Passed locally with Java 21 on the
branch-4.1backport:Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings;
git diff --checkalso passed. LevelDB-backed variants were not run on Apple Silicon.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport.