Repository navigation
Remove stale dev/diffs test exclusions whose tracking issues are closed #5569
Description
Three exclusions in dev/diffs/*.diff cite Comet issues that have since been fixed, so they now suppress Spark SQL test coverage for no reason. Each is present in more than one version diff, so they should be removed across all affected diffs in one pass rather than in whichever version PR happens to touch them next: new version diffs are seeded from the previous version's, so a partial fix diverges the diffs and the next reseed reintroduces the stale entry.
Surfaced in review of #4950 by @sunchao.
1. Variant shredding suites
VariantShreddingSuite and ParquetVariantShreddingSuite are given an IgnoreCometSuite mixin, which skips all 15 tests when ENABLE_COMET=true.
- Cited issue: [native_iceberg_compat]
VariantShreddingSuitetest failures with Spark 4.0.0 #2209, closed by merged PR fix: fall back for shredded Variant scans on Spark 4.0 #4084. - fix: fall back for shredded Variant scans on Spark 4.0 #4084 added the shredded-Variant scan fallback and already removed these same exclusions from
dev/diffs/4.0.4.diff(it has zero occurrences ofVariantShreddingSuite);4.1.3.diffwas never cleaned up. - Affected:
dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff. - Removing the mixins should also drop the now-unused
IgnoreCometSuiteimport in each file.
This is the highest-value one: it drops regression coverage for the silent-null corruption that #4084 fixed.
2. having-and-order-by-recursive-type-name-resolution.sql
The input file carries a --SET spark.comet.enabled = false header, which runs all 45 queries without Comet.
- Cited issue: Comet native sort lacks row-format support for Struct(Map(...)) sort keys #4123, closed by merged PR fix: prevent native sort crash for Struct(Map(...)) keys #4157, which added recursive fallback for map-containing aggregate keys plus a regression test for the exact
Struct(Map(...))HAVING / ORDER BY shape. - Affected:
dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff.
3. Parquet V2 writer matrix in ParquetTypeWideningSuite
withAllParquetWriters { ... } is commented out, leaving all 82 generated widening cases on Parquet's default V1 writer.
- Cited issue: Support DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY #574, closed: the active
native_datafusionandnative_iceberg_compatscans supportDELTA_BINARY_PACKED,DELTA_BYTE_ARRAYandDELTA_LENGTH_BYTE_ARRAY. - The related
SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded stringstest inParquetIOSuiteis still taggedIgnoreComet("Comet doesn't support DELTA encoding yet")and should be re-enabled with it. - Affected:
dev/diffs/4.0.4.diff,dev/diffs/4.1.3.diff,dev/diffs/4.2.0.diff(byte-identical in all three). - If the legacy
native_cometscan still cannot read these encodings, scope the exclusion to that scan implementation rather than removing the whole matrix.
Suggested approach
One PR per item, each removing the exclusion from every diff that carries it, verified by the corresponding Spark SQL Tests CI job for each affected version. Where a specific case still fails, replace the blanket exclusion with a case-scoped IgnoreComet linked to a current issue, per the rule in docs/source/contributor-guide/adding_a_new_spark_version.md ("do not disable whole suites").
Activity
rich7420 commentedon Sep 7, 2026
take
andygrove commentedon Sep 22, 2026
All three items are now fixed, though not in the one-pass-per-item shape this issue proposed.
having-and-order-by-recursive-type-name-resolution.sqlDELTA_LENGTH_BYTE_ARRAYThe 4.0 and 4.1 halves landed on main between 2026-09-08 and 2026-09-09. The 4.2 half is in #4950 and is not yet merged, so this issue should close with that PR rather than now.
One thing this issue predicted and that turned out to be worth recording: the reseed hazard is real, but it does not need a reseed to bite.
dev/diffs/4.2.0.diffexists only on the #4950 branch, so none of #5745, #5755 or #5760 reached it, and merging main into that branch produced no conflict at any point -- the 4.2 diff simply sat at its 2026-08-31 state while the sibling diffs moved on. By the time it was checked, seven upstream commits to4.1.3.diffhad accumulated. The general fix is to not leave a supported version's suite outside the automated tiers: #4950 now also moves Spark 4.2 into the nightly tier, since a diff is kept current as a side effect of its suite running.dwsmith1983 commentedon Oct 4, 2026
#4950 has merged now, so all three exclusions are gone from every version diff: #5745, #5755 and #5760 cleaned up 4.0 and 4.1, and #4950 added
4.2.0.diffwithout them. The only Variant exclusion left is one test inParquetVariantShreddingSuite, ignored for #5741, which is a separate issue. I think this can be closed.rich7420 commentedon Oct 16, 2026
@andygrove hi , could you help me to close this?
andygrove commentedon Oct 16, 2026
Thank you @dwsmith1983 and @rich7420
rich7420 commentedon Oct 16, 2026
thanks for your help @andygrove !