Repository navigation
[SPARK-59747][CORE] Don't retry compaction of rolled event log files with malformed input - #58996
DengYiping wants to merge 4 commits into
Conversation
654dd3b to
7b88f62
Compare
…with malformed input When the rolled event log files being compacted contain bytes that are not valid UTF-8, EventLogFileCompactor fails with a CharacterCodingException. FsHistoryProvider.compact() only records lastEvaluatedForCompaction after a successful compaction, so the same unchanged files are replayed and the failure is logged at ERROR every time the last event log file grows. Catch CharacterCodingException around the compaction, log it at WARN, and still record the index as evaluated so that compaction is not retried until a new event log file is rolled.
7b88f62 to
4b172e3
Compare
vboo123
left a comment
There was a problem hiding this comment.
Read through this alongside EventLogFileCompactor — the decode failure happens in the first-pass replay (initializeBuilders), before rewrite() creates the compact file or the originals get deleted, so skipping the index leaves no half-compacted state behind. And since the rolled files are immutable, a retry would just fail the same way, so recording lastEvaluatedForCompaction and moving on makes sense to me. LGTM.
However, I did have a question for my own learning. How do the malformed bytes end up in an already-rolled file in the first place? Is it something like a driver killed mid-write leaving a truncated multi-byte char, or another case you've hit? From what I can tell we'd only run into this with compaction enabled (maxFilesToRetain lowered from its default) and a corrupted rolled file, right?
|
Thanks for the review! Honestly, we haven't pinned down the writer yet. We hit this in production on a Spark 4.0 History Server: tens of thousands of And yes, your reading is right: it needs compaction enabled ( |
uros-b
left a comment
There was a problem hiding this comment.
Thank you @DengYiping and @vboo123! The fix seems minimal and correct
Adding @HeartSaVioR - original author of the event-log compaction feature (EventLogFileCompactor / SPARK-29779, SPARK-30481)
|
@HeartSaVioR, gentle follow-up: would you have a chance to review this small event-log compaction fix when you have time? It avoids repeatedly retrying compaction of unchanged rolled files with malformed input. Thanks to everyone who has reviewed so far, and thank you for taking a look! |
sarutak
left a comment
There was a problem hiding this comment.
Hi @DengYiping, thank you for fixing this issue.
Two small, non-blocking suggestions, both about resilience rather than behavior:
-
The test depends on listing replay halting right after the environment update (so the appended malformed bytes are read only during compaction's full replay, not during listing). That halt relies on
createTestConf()defaultingFAST_IN_PROGRESS_PARSINGto true together with the app being in-progress (the writer is not stopped beforeupdateAndCheck). SettingFAST_IN_PROGRESS_PARSINGexplicitly in the test, plus a one-line comment noting that listing halts after the environment update while compaction replays the file in full, would make the padding (the 1000JobStartevents) self-explanatory and keep the test from silently changing meaning if that default ever changes. -
The safety of catching
CharacterCodingExceptionaround the wholecompactor.compact(...)call rests on the invariant that phase 1 (initializeBuilders) decodes every file in full and therefore always throws before phase 2 (rewrite()) mutates on-disk state. A short comment (or a test assertion) recording that invariant would guard against a future regression if an early-haltingEventFilterBuilderis ever added, which could otherwise let the throw happen after a partial compact file is written.
|
Thanks @sarutak! Both suggestions addressed. Could you take another look? |
338eabf to
872822d
Compare
|
I almost forgot about this stuff and I didn't use this for 5+ years, so if other committers+ can help review, that's good. |
|
@viirya to see whether we have redundant/duplicate/correlated effort since he have been dealing with event log. |
viirya
left a comment
There was a problem hiding this comment.
Thanks for the fix. The safety argument holds: an invalid UTF-8 sequence always fails during the initializeBuilders replay, which reads every file to compact to the end before rewrite() creates anything, and rewrite() decodes strictly as well, so a failure here leaves no partial compact file behind. Recording the index as evaluated is consistent with how LOW_SCORE_FOR_COMPACTION is handled.
For context, #59328 (SPARK-60110) makes EventLogFileCompactor.compact return a new INCOMPLETE_REPLAY code when replay does not fully read the files to compact, and relies on the same "re-evaluate on the next rolled file" behavior through lastEvaluatedForCompaction. This PR covers the case where that replay throws instead. The two don't conflict, and this PR doesn't need to wait for #59328. Once both are in, I plan to follow up by moving this handling into initializeBuilders so the policy lives in one place. A couple of small comments inline.
| compactor.compact(reader.listEventLogFiles) | ||
| } catch { | ||
| case e: CharacterCodingException => | ||
| // This relies on initializeBuilders decoding every input file in full before |
There was a problem hiding this comment.
With #59328, initializeBuilders stops at the first file that is not fully replayed, so "decoding every input file in full" will no longer hold literally, even though the safety argument still does. Could we phrase it in terms of what this actually relies on? Something like:
// rewrite() only runs after initializeBuilders has replayed every file to compact
// without error, so no files have been written or deleted at this point.| // Only rolled event log files are compacted and they are no longer modified, | ||
| // so retrying would fail the same way. Record this index as evaluated to | ||
| // avoid replaying the same files on every scan. | ||
| logWarning(log"Skipping compaction for ${MDC(PATH, rootPath)} as the event " + |
There was a problem hiding this comment.
The exception message here is just Input length = 1, and ReplayListenerBus rethrows IOException without logging the source, so this WARN only identifies the application directory, not which rolled file is corrupted. Since the same file also breaks loading the application UI (parseAppEventLogs doesn't catch this either), it would help operators to know which file to look at. Is there a simple way to include it here, or would it be easier to log it where the file is known?
|
Thanks @viirya! Both comments addressed |
|
@DengYiping
|
Done, thanks for the feedback |
…with malformed input ### What changes were proposed in this pull request? In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it. ``` build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #58996 from DengYiping/shs-compaction-malformed-input. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org> (cherry picked from commit d9a35e9) Signed-off-by: Kousuke Saruta <sarutak@apache.org>
…with malformed input ### What changes were proposed in this pull request? In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it. ``` build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #58996 from DengYiping/shs-compaction-malformed-input. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org> (cherry picked from commit d9a35e9) Signed-off-by: Kousuke Saruta <sarutak@apache.org>
…with malformed input ### What changes were proposed in this pull request? In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it. ``` build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #58996 from DengYiping/shs-compaction-malformed-input. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org> (cherry picked from commit d9a35e9) Signed-off-by: Kousuke Saruta <sarutak@apache.org>
|
Hi @DengYiping this change conflicts with |
Will do. Opening a back-porting PR now ... |
…iles with malformed input ### What changes were proposed in this pull request? Backport #58996 to `branch-4.0`. In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. The conflict resolution retains this branch's existing single `fileCompactor`. The test aliases NIO `Files` as `JavaFiles` to preserve existing Guava `Files` calls. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. Passed locally with Java 21 on the `branch-4.0` backport: ```sh build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \ 'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \ 'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite' ``` Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport. Closes #59620 from DengYiping/ydeng/spark-59747-branch-4.0. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org>
…iles with malformed input ### What changes were proposed in this pull request? Backport #58996 to `branch-4.1`. In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index. `EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception. The conflict resolution retains this branch's existing single `fileCompactor`. ### Why are the changes needed? When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR. The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed. With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before. ### Does this PR introduce _any_ user-facing change? Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before. ### How was this patch tested? Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. Passed locally with Java 21 on the `branch-4.1` backport: ```sh build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \ 'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \ 'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite' ``` Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport. Closes #59619 from DengYiping/ydeng/spark-59747-branch-4.1. Authored-by: Yiping Deng <yiping.deng.98@gmail.com> Signed-off-by: Kousuke Saruta <sarutak@apache.org>
What changes were proposed in this pull request?
In
FsHistoryProvider.compact(), catchCharacterCodingExceptionthrown while compacting rolled event log files, log it at WARN, and still recordlastEvaluatedForCompactionfor the current index.EventLogFileCompactor.initializeBuildersalso logs a WARN naming the malformed event log file before rethrowing the decoding exception.Why are the changes needed?
When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8,
EventLogFileCompactorfails with aMalformedInputExceptionwhile replaying them to initialize the event filter builders.compact()only recordslastEvaluatedForCompactionafter compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.
With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.
Does this PR introduce any user-facing change?
Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.
How was this patch tested?
Added a new test to
FsHistoryProviderSuite. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created,lastEvaluatedForCompactionis recorded, and a WARN names the malformed file. The test fails without this change (None did not equal Some(2), withMalformedInputExceptionlogged at ERROR) and passes with it.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)