Repository navigation
Skip the Spark RocksDB state-store suites when Comet is enabled #5986
Description
What is the problem the feature request solves?
The Spark SQL CI matrix runs sql/testOnly * (see dev/ci/spark-sql-modules.py), so every
RocksDB state-store suite under
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/ runs with Comet
enabled. None of them exercise a Comet code path:
CometExecRuleandCometScanRuleboth bail out at the top of_applyfor any streaming
plan (ShimCometStreaming.isStreamingPlan), so the suites that drive streaming queries
(RocksDBStateStoreIntegrationSuite,RocksDBStateStoreCheckpointFormatV2Suite,
RocksDBCheckpointFailureInjectionSuite) execute plain Spark from end to end.- The rest (
RocksDBSuite,RocksDBStateStoreSuite,RocksDBLineageSuite) call
StateStore/RocksDBAPIs directly and never build a query plan at all.
Current state, per dev/diffs/:
| Suite | 3.4.3 | 3.5.9 | 4.0.4 | 4.1.3 | shard |
|---|---|---|---|---|---|
RocksDBStateStoreIntegrationSuite |
runs | runs | runs | ignored | sql_core-1 |
RocksDBStateStoreSuite |
runs | runs | runs | runs | sql_core-2 |
RocksDBSuite |
runs | runs | runs | runs | sql_core-3 |
RocksDBStateStoreCheckpointFormatV2Suite |
n/a | n/a | runs | runs | sql_core-1 |
RocksDBCheckpointFailureInjectionSuite |
n/a | n/a | n/a | runs | sql_core-3 |
RocksDBLineageSuite |
n/a | n/a | n/a | runs | sql_core-1 |
So one suite out of six is skipped, on one Spark version out of four, and the asymmetry looks
accidental rather than deliberate.
There is already precedent for skipping this family. StateStoreSuite -- the
StateStoreSuiteBase[HDFSBackedStateStoreProvider] twin of RocksDBStateStoreSuite -- is
skipped under Comet on 4.0 and 4.1, with the comment "The tests target streaming state-store
internals (StateStore.get/put/commit), not SQL execution paths, and the maintenance test is
flaky in CI" (#4221). The same reasoning applies verbatim to the RocksDB provider, which is the
heavier of the two.
These suites are not cheap. RocksDBSuite is ~4000 lines and @SlowSQLTest,
RocksDBStateStoreSuite is @ExtendedSQLTest, and AlsoTestWithRocksDBFeatures /
AlsoTestWithEncodingTypes multiply each test case across changelog-checkpointing, column-family
and encoding permutations. They land in all three sql_core shards, which are the rows already
tuned for the 7 GB runner budget.
Describe the potential solution
Mix IgnoreCometSuite into the RocksDB state-store suites in all four dev/diffs/*.diff files,
so the family is skipped consistently rather than on one version only.
Leave RocksDBStateStoreLockHardeningSuite and RocksDBStateEncoderSuite running: they are
plain SparkFunSuites that never start a SparkSession, so Comet is never loaded and skipping
them would only cost upstream regression detection for nothing.
Additional context
No response
Activity
test: skip the Spark RocksDB state-store suites when Comet is enabled (…