Worker autoscaling does not effectively handle bursty concurrent queries on Iceberg data

[saksham295]

Trino version

481

Please describe the bug

Description

We are running Trino on Kubernetes and are experiencing high query latency when our backend service sends a burst of concurrent requests.

Our setup consists of:

• 1 Coordinator: 2 CPU / 16 GiB memory
• 2 Workers: 4 CPU / 16 GiB memory each
• Kubernetes HPA configured for the workers
• Apache Iceberg as the table format
• Data stored as Parquet files
• Backend API timeout: 60 seconds

Each API request queries data from a different partition of the Iceberg table. Consequently, concurrent API requests generally result in Trino reading different partitions and therefore different Parquet files.

The approximate size and schema of the two tables being joined in the query are:

Table	Data size	Number of columns
Table 1	894.7 MB	211
Table 2	341.6 MB	30

For the query under the concurrent workload, Trino reports approximately:

Metric	Value
Input rows	~6 million
Input data	~500 MB
Physical input rows	~4.29 million
Physical input data	~34 MB
Output rows	11
Output data	~2.15 KB

Problem

Under normal traffic, query latency is acceptable. However, when approximately 50 API requests are made concurrently, the Trino query portion of the requests takes approximately 30–40 seconds.

Since the API timeout is 60 seconds, many requests eventually time out instead of returning a response.

Autoscaling behavior

We currently have a Kubernetes HPA configured on the Trino workers.

However, during a sudden burst of requests, all queries arrive at approximately the same time.

The existing workers begin processing the queries concurrently, and the workload becomes saturated. The HPA does not appear to add workers early enough to prevent the increase in query latency.

As a result, queries experience high latency before additional workers become available, causing many API requests to approach or exceed the 60-second timeout.

Expected behavior

We would like to understand the recommended approach for elastic Trino deployments on Kubernetes for bursty, highly concurrent Iceberg workloads.

Specifically, we would like to know whether Trino provides a mechanism or recommended metric that can be used to scale workers based on query demand rather than only CPU utilization.

Potential signals could include:

• Queued queries
• Running queries
• Pending tasks
• Task backlog
• Worker utilization
• Query concurrency
• Number of active splits
• Other Trino metrics that indicate insufficient worker capacity

Environment

Deployment:
Coordinator: 1 × 2 CPU / 16 GiB
Workers: 2 × 4 CPU / 16 GiB

Autoscaling:
Kubernetes HPA

Storage/Table format:
Apache Iceberg
Parquet

Workload:
Backend API → Trino → Iceberg/Parquet
~50 concurrent API requests during peak burst

API timeout:
60 seconds

Observed Trino latency:
~30–40 seconds under ~50 concurrent requests

Data access pattern:
Concurrent requests generally target different Iceberg partitions / Parquet files.

Table 1:
Data size: ~894.7 MB
Columns: 211

Table 2:
Data size: ~341.6 MB
Columns: 30

Query metrics:
Input rows: ~6 million
Input data: ~500 MB
Physical input rows: ~4.29 million
Physical input data: ~34 MB
Output rows: 11
Output data: ~2.15 KB

We are primarily looking for guidance on whether our current architecture/configuration is appropriate for this workload, and what the recommended approach is for scaling Trino workers quickly enough to handle sudden query bursts without causing API-level timeouts.

Additional information

We can provide query plans, Trino UI screenshots, worker resource utilization, HPA configuration, and other metrics from the concurrent-load test if they would be useful for diagnosing the issue.