Skip to content

[SPARK-59747][CORE] Don't retry compaction of rolled event log files with malformed input - #58996

Closed
DengYiping wants to merge 4 commits into
apache:masterfrom
DengYiping:shs-compaction-malformed-input
Closed

DengYiping wants to merge 4 commits into
apache:masterfrom
DengYiping:shs-compaction-malformed-input

Conversation

@DengYiping

@DengYiping DengYiping commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

In FsHistoryProvider.compact(), catch CharacterCodingException thrown while compacting rolled event log files, log it at WARN, and still record lastEvaluatedForCompaction for the current index.

EventLogFileCompactor.initializeBuilders also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, EventLogFileCompactor fails with a MalformedInputException while replaying them to initialize the event filter builders. compact() only records lastEvaluatedForCompaction after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

Does this PR introduce any user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

How was this patch tested?

Added a new test to FsHistoryProviderSuite. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, lastEvaluatedForCompaction is recorded, and a WARN names the malformed file. The test fails without this change (None did not equal Some(2), with MalformedInputException logged at ERROR) and passes with it.

build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

@DengYiping
DengYiping force-pushed the shs-compaction-malformed-input branch 2 times, most recently from 654dd3b to 7b88f62 Compare September 23, 2026 13:35
…with malformed input

When the rolled event log files being compacted contain bytes that are not valid UTF-8,
EventLogFileCompactor fails with a CharacterCodingException. FsHistoryProvider.compact() only
records lastEvaluatedForCompaction after a successful compaction, so the same unchanged files
are replayed and the failure is logged at ERROR every time the last event log file grows.

Catch CharacterCodingException around the compaction, log it at WARN, and still record the
index as evaluated so that compaction is not retried until a new event log file is rolled.
@DengYiping
DengYiping force-pushed the shs-compaction-malformed-input branch from 7b88f62 to 4b172e3 Compare September 23, 2026 13:56

@vboo123 vboo123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read through this alongside EventLogFileCompactor — the decode failure happens in the first-pass replay (initializeBuilders), before rewrite() creates the compact file or the originals get deleted, so skipping the index leaves no half-compacted state behind. And since the rolled files are immutable, a retry would just fail the same way, so recording lastEvaluatedForCompaction and moving on makes sense to me. LGTM.

However, I did have a question for my own learning. How do the malformed bytes end up in an already-rolled file in the first place? Is it something like a driver killed mid-write leaving a truncated multi-byte char, or another case you've hit? From what I can tell we'd only run into this with compaction enabled (maxFilesToRetain lowered from its default) and a corrupted rolled file, right?

@DengYiping

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Honestly, we haven't pinned down the writer yet. We hit this in production on a Spark 4.0 History Server: tens of thousands of MalformedInputExceptions from compaction over about ten days, all from event logs in a single bucket, while other regions with the same setup were clean. In our setup the event logs are written straight to S3 through S3A, uncompressed, with rolling enabled. S3A only makes an object visible on close, so a driver killed mid-write shouldn't leave a truncated rolled file, and the in-progress file is never compacted anyway. Spark's own writer goes through OutputStreamWriter(UTF_8), which replaces rather than emits invalid sequences, so we suspect something outside Spark's writer, but we haven't confirmed it.

And yes, your reading is right: it needs compaction enabled (spark.history.fs.eventLog.rolling.maxFilesToRetain lowered from its Int.MaxValue default; we run with 5), rolling event logs, and malformed bytes in a rolled file somewhere the listing pass doesn't reach. The listing replay stops once it has the app-start and environment events and then only re-reads the tail of the last file, so bytes further into an older rolled file are only ever decoded by compaction. That's the retry loop this PR stops.

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @DengYiping and @vboo123! The fix seems minimal and correct

Adding @HeartSaVioR - original author of the event-log compaction feature (EventLogFileCompactor / SPARK-29779, SPARK-30481)

@uros-b
uros-b requested a review from HeartSaVioR September 29, 2026 19:47
@DengYiping

Copy link
Copy Markdown
Contributor Author

@HeartSaVioR, gentle follow-up: would you have a chance to review this small event-log compaction fix when you have time? It avoids repeatedly retrying compaction of unchanged rolled files with malformed input. Thanks to everyone who has reviewed so far, and thank you for taking a look!

@uros-b
uros-b requested a review from sarutak October 12, 2026 10:09

@sarutak sarutak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @DengYiping, thank you for fixing this issue.

Two small, non-blocking suggestions, both about resilience rather than behavior:

  1. The test depends on listing replay halting right after the environment update (so the appended malformed bytes are read only during compaction's full replay, not during listing). That halt relies on createTestConf() defaulting FAST_IN_PROGRESS_PARSING to true together with the app being in-progress (the writer is not stopped before updateAndCheck). Setting FAST_IN_PROGRESS_PARSING explicitly in the test, plus a one-line comment noting that listing halts after the environment update while compaction replays the file in full, would make the padding (the 1000 JobStart events) self-explanatory and keep the test from silently changing meaning if that default ever changes.

  2. The safety of catching CharacterCodingException around the whole compactor.compact(...) call rests on the invariant that phase 1 (initializeBuilders) decodes every file in full and therefore always throws before phase 2 (rewrite()) mutates on-disk state. A short comment (or a test assertion) recording that invariant would guard against a future regression if an early-halting EventFilterBuilder is ever added, which could otherwise let the throw happen after a partial compact file is written.

@DengYiping

Copy link
Copy Markdown
Contributor Author

Thanks @sarutak! Both suggestions addressed. Could you take another look?

@DengYiping
DengYiping force-pushed the shs-compaction-malformed-input branch from 338eabf to 872822d Compare October 13, 2026 13:59
@HeartSaVioR

Copy link
Copy Markdown
Contributor

I almost forgot about this stuff and I didn't use this for 5+ years, so if other committers+ can help review, that's good.

@HeartSaVioR

Copy link
Copy Markdown
Contributor

@viirya to see whether we have redundant/duplicate/correlated effort since he have been dealing with event log.

@viirya viirya left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. The safety argument holds: an invalid UTF-8 sequence always fails during the initializeBuilders replay, which reads every file to compact to the end before rewrite() creates anything, and rewrite() decodes strictly as well, so a failure here leaves no partial compact file behind. Recording the index as evaluated is consistent with how LOW_SCORE_FOR_COMPACTION is handled.

For context, #59328 (SPARK-60110) makes EventLogFileCompactor.compact return a new INCOMPLETE_REPLAY code when replay does not fully read the files to compact, and relies on the same "re-evaluate on the next rolled file" behavior through lastEvaluatedForCompaction. This PR covers the case where that replay throws instead. The two don't conflict, and this PR doesn't need to wait for #59328. Once both are in, I plan to follow up by moving this handling into initializeBuilders so the policy lives in one place. A couple of small comments inline.

compactor.compact(reader.listEventLogFiles)
} catch {
case e: CharacterCodingException =>
// This relies on initializeBuilders decoding every input file in full before

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With #59328, initializeBuilders stops at the first file that is not fully replayed, so "decoding every input file in full" will no longer hold literally, even though the safety argument still does. Could we phrase it in terms of what this actually relies on? Something like:

// rewrite() only runs after initializeBuilders has replayed every file to compact
// without error, so no files have been written or deleted at this point.

// Only rolled event log files are compacted and they are no longer modified,
// so retrying would fail the same way. Record this index as evaluated to
// avoid replaying the same files on every scan.
logWarning(log"Skipping compaction for ${MDC(PATH, rootPath)} as the event " +

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exception message here is just Input length = 1, and ReplayListenerBus rethrows IOException without logging the source, so this WARN only identifies the application directory, not which rolled file is corrupted. Since the same file also breaks loading the application UI (parseAppEventLogs doesn't catch this either), it would help operators to know which file to look at. Is there a simple way to include it here, or would it be easier to log it where the file is known?

@DengYiping

DengYiping commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @viirya! Both comments addressed

@sarutak

sarutak commented Oct 15, 2026

Copy link
Copy Markdown
Member

@DengYiping
One small request before merge: could you update the PR description to match 0a2d73a? It currently describes only the change in FsHistoryProvider.compact().

  1. "What changes were proposed": also mention that EventLogFileCompactor.initializeBuilders now logs a WARN naming the malformed event log file before rethrowing.
  2. "Does this PR introduce any user-facing change": the History Server now logs the path of the malformed file in addition to the WARN for the application directory.
  3. "How was this patch tested": the test now also checks that the WARN names the malformed file.

@DengYiping

Copy link
Copy Markdown
Contributor Author

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

Done, thanks for the feedback

@sarutak sarutak closed this in d9a35e9 Oct 20, 2026
sarutak pushed a commit that referenced this pull request Oct 20, 2026
…with malformed input

### What changes were proposed in this pull request?

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it.

```
build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history
```

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

Closes #58996 from DengYiping/shs-compaction-malformed-input.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
(cherry picked from commit d9a35e9)
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
sarutak pushed a commit that referenced this pull request Oct 20, 2026
…with malformed input

### What changes were proposed in this pull request?

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it.

```
build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history
```

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

Closes #58996 from DengYiping/shs-compaction-malformed-input.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
(cherry picked from commit d9a35e9)
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
sarutak pushed a commit that referenced this pull request Oct 20, 2026
…with malformed input

### What changes were proposed in this pull request?

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file. The test fails without this change (`None did not equal Some(2)`, with `MalformedInputException` logged at ERROR) and passes with it.

```
build/mvn -pl core -am test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history
```

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

Closes #58996 from DengYiping/shs-compaction-malformed-input.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
(cherry picked from commit d9a35e9)
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
@sarutak

sarutak commented Oct 20, 2026

Copy link
Copy Markdown
Member

Merge Summary:

Posted by merge_spark_pr.py

@sarutak

sarutak commented Oct 20, 2026

Copy link
Copy Markdown
Member

Hi @DengYiping this change conflicts with branch-4.1 and branch-4.0. Could you open backport-PRs for them?

@DengYiping

Copy link
Copy Markdown
Contributor Author

Hi @DengYiping this change conflicts with branch-4.1 and branch-4.0. Could you open backport-PRs for them?

Will do. Opening a back-porting PR now ...

sarutak pushed a commit that referenced this pull request Oct 27, 2026
…iles with malformed input

### What changes were proposed in this pull request?

Backport #58996 to `branch-4.0`.

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

The conflict resolution retains this branch's existing single `fileCompactor`. The test aliases NIO `Files` as `JavaFiles` to preserve existing Guava `Files` calls.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file.

Passed locally with Java 21 on the `branch-4.0` backport:

```sh
build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \
  'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \
  'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite'
```

Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport.

Closes #59620 from DengYiping/ydeng/spark-59747-branch-4.0.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
sarutak pushed a commit that referenced this pull request Oct 28, 2026
…iles with malformed input

### What changes were proposed in this pull request?

Backport #58996 to `branch-4.1`.

In `FsHistoryProvider.compact()`, catch `CharacterCodingException` thrown while compacting rolled event log files, log it at WARN, and still record `lastEvaluatedForCompaction` for the current index.

`EventLogFileCompactor.initializeBuilders` also logs a WARN naming the malformed event log file before rethrowing the decoding exception.

The conflict resolution retains this branch's existing single `fileCompactor`.

### Why are the changes needed?

When the rolled event log files that are eligible for compaction contain bytes that are not valid UTF-8, `EventLogFileCompactor` fails with a `MalformedInputException` while replaying them to initialize the event filter builders. `compact()` only records `lastEvaluatedForCompaction` after compaction succeeds, and the exception reaches the generic handler, which logs it at ERROR.

The listing replay stops early, so it never reaches the malformed bytes and the application keeps being listed. Each time the last (non-compacted) event log file grows, the log is reprocessed and compaction of the same unchanged rolled files is tried again. For a long-running application this logs a full stack trace at ERROR on every scan until a new event log file is rolled. Rolled files are no longer modified, so the retry can never succeed.

With this change, compaction for that index is skipped, and it is only tried again once a new event log file is rolled. Other failures are handled as before.

### Does this PR introduce _any_ user-facing change?

Yes, but only in History Server logs. When compaction fails because the event log files contain malformed input, the History Server logs a WARN once per rolled event log index instead of an ERROR on every scan. An additional WARN identifies the path of the malformed event log file, alongside the WARN for the application directory. The application listing and replay are unchanged, and the event log files are left uncompacted as before.

### How was this patch tested?

Added a new test to `FsHistoryProviderSuite`. It writes a rolled event log file with a malformed UTF-8 sequence after the events used for listing, then checks that the application is still listed, no compact file is created, `lastEvaluatedForCompaction` is recorded, and a WARN names the malformed file.

Passed locally with Java 21 on the `branch-4.1` backport:

```sh
build/sbt 'core/Compile/scalastyle' 'core/Test/scalastyle' \
  'core/testOnly org.apache.spark.deploy.history.RocksDBBackendFsHistoryProviderSuite org.apache.spark.deploy.history.RocksDBBackendWithProtobufSerializerSuite -- -z "SPARK-59747"' \
  'core/testOnly org.apache.spark.deploy.history.EventLogFileCompactorSuite'
```

Both malformed-input regression variants passed (2 tests), including the WARN naming the malformed file. All 12 compactor tests passed, including the SPARK-60110 incomplete-replay coverage. Both style checks passed with zero errors/warnings; `git diff --check` also passed. LevelDB-backed variants were not run on Apple Silicon.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5) for the original patch; OpenAI Codex (GPT-5) for the backport.

Closes #59619 from DengYiping/ydeng/spark-59747-branch-4.1.

Authored-by: Yiping Deng <yiping.deng.98@gmail.com>
Signed-off-by: Kousuke Saruta <sarutak@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants