Repository navigation
Inconsistency in the latency breakdown measurement #20
Description
Activity
The data transfer between CPU and GPU can be executed in parallel.
In our code we put pq computation and kv transfer in different cuda stream such that the computation and KV transfer are overlapped. Since the pq_start and pq_end cuda event is recorded in "computation stream", theoretically the measurement result doesn't include the communication latency.
According to Fig. 12(b), the PQ computation time includes top-k token index time, without the communication latency of top-k tokens' kv from host CPU. However, in the pq_search.py file, the pq_start and pq_end functions profile the entire process including the communication latency (cache_managers[self.rank].fetch_and_concat_kv_w_cache), which contradicts what the paper said. Could you please explain on this?