Trino version
478 (applies to 465+; baseline comparison against 443)
Problem
Since #23945 each Iceberg split source produces batches on its own thread, and each active scan keeps 2 × availableProcessors manifest-decode tasks in flight in the shared iceberg-split-manager pool (strict FIFO, no per-query fairness, default-capped at 32 threads since #25717). With large manifests (ours: ~12 MB, 1.5–3 s to decode one), a small query's first split batch queues behind dozens of seconds-long tasks from one big query — its scan stage sits in PLANNED for 10–60 s. Not CPU-bound (reproduces with coordinator CPU at 45%); non-Iceberg queries unaffected. Pre-465, sequential batch production bounded each query's in-flight contribution and small queries passed in ~1 decode round.
Reproduction
One table whose snapshots carry hundreds of multi-MB manifests (ours is a wide event-log table where each commit adds thousands of files, so every manifest holds thousands of entries with full column stats), plus one small table. Any table with a few hundred manifests of a few MB each reproduces; we can share a generator script if useful. Run a single 3-way self-join on the big table; probe the small table (WHERE pk = ? LIMIT 1, ~20 splits) every 15 s. Same hardware, pools 40/20:
|
443 |
478 |
| big query wall |
613 s |
605 s |
| probe p50 / p95 |
0.9 / 2.2 s |
10.2 / 12.5 s |
Thread dumps: 443 enumerates inline on the query's scheduler thread (≤1 scan at a time); 478 shows up to 18 iceberg-split-source-* threads parked in ParallelIterable.hasNext. Decoding the same manifests via "t$files" on workers takes 0.6 s at the same moment, so storage/decode cost is not the issue.
Single-variable proof
For diagnosis we patched a session toggle that swaps the split-source executor for directExecutor() (restoring pre-465 sequential consumption; shared pool untouched). Same storm, paired probes: big query async → probes 17.7 / 62.5 s (p50/p95); big query sync → 1.2 / 2.3 s — our 443 baseline. Toggling only the probe's session changes nothing, so the big scans fill the queue. Cost: largest scan ~20–50% slower, mid-size queries ~40% faster, total wall for 3 concurrent bigs unchanged.
Question
Is there a recommended way to mitigate this on current releases, or are there plans to address the interaction (e.g., bounding a query's concurrent in-flight decode work, or fairness in the shared pool)? We're happy to provide the full reproduction and measurements if helpful.
Trino version
478 (applies to 465+; baseline comparison against 443)
Problem
Since #23945 each Iceberg split source produces batches on its own thread, and each active scan keeps
2 × availableProcessorsmanifest-decode tasks in flight in the sharediceberg-split-managerpool (strict FIFO, no per-query fairness, default-capped at 32 threads since #25717). With large manifests (ours: ~12 MB, 1.5–3 s to decode one), a small query's first split batch queues behind dozens of seconds-long tasks from one big query — its scan stage sits in PLANNED for 10–60 s. Not CPU-bound (reproduces with coordinator CPU at 45%); non-Iceberg queries unaffected. Pre-465, sequential batch production bounded each query's in-flight contribution and small queries passed in ~1 decode round.Reproduction
One table whose snapshots carry hundreds of multi-MB manifests (ours is a wide event-log table where each commit adds thousands of files, so every manifest holds thousands of entries with full column stats), plus one small table. Any table with a few hundred manifests of a few MB each reproduces; we can share a generator script if useful. Run a single 3-way self-join on the big table; probe the small table (
WHERE pk = ? LIMIT 1, ~20 splits) every 15 s. Same hardware, pools 40/20:Thread dumps: 443 enumerates inline on the query's scheduler thread (≤1 scan at a time); 478 shows up to 18
iceberg-split-source-*threads parked inParallelIterable.hasNext. Decoding the same manifests via"t$files"on workers takes 0.6 s at the same moment, so storage/decode cost is not the issue.Single-variable proof
For diagnosis we patched a session toggle that swaps the split-source executor for
directExecutor()(restoring pre-465 sequential consumption; shared pool untouched). Same storm, paired probes: big query async → probes 17.7 / 62.5 s (p50/p95); big query sync → 1.2 / 2.3 s — our 443 baseline. Toggling only the probe's session changes nothing, so the big scans fill the queue. Cost: largest scan ~20–50% slower, mid-size queries ~40% faster, total wall for 3 concurrent bigs unchanged.Question
Is there a recommended way to mitigate this on current releases, or are there plans to address the interaction (e.g., bounding a query's concurrent in-flight decode work, or fairness in the shared pool)? We're happy to provide the full reproduction and measurements if helpful.