This repository was archived by the owner on Oct 22, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Oct 22, 2026. It is now read-only.
[BUG] TiSpark recent release is slower than before #2105
Description
The problem is located that in the recent release of TiSpark, the default value of PARTITION_PER_SPLIT is changed from 10 to 1. It results in increasing Spark tasks.
some related problems:
- ScanRequest receive a slower response from tikv when scanning meta data
- cause: scanning is not concurrent.
- Memory usage incresed.
- cause: the memory usage of ColumnVector should be optimized.
Affected versions: v2.3.14 to v2.3.16, v2.4.1
Activity
crabo commentedon May 11, 2022
"Memory usage": Pls also check DAGIterator.process(), the underlying grpc always thow OutOfDirectMemoryError even in unpooled mode. As 10Million rows table scan in ETL, Off-Heap mem is requried roughly 5GB, that's really a big waste.