Morsel-Driven Parallelism Using Rayon #2199

Activity

1. tustvold

   added

   enhancementNew feature or requestNew feature or request

   on Apr 11, 2022
2. xudong963

   added

   design

   on Apr 11, 2022
3. tustvold

   added 8 commits that reference this issue on Apr 13, 2022

   Morsel-driven Parallelism using rayon (apache#2199)

   f8da884

   Morsel-driven Parallelism using rayon (apache#2199)

   56702d8

   Morsel-driven Parallelism using rayon (apache#2199)

   02f4d95

   Morsel-driven Parallelism using rayon (apache#2199)

   35d84ca

   Morsel-driven Parallelism using rayon (apache#2199)

   ac60247

   Morsel-driven Parallelism using rayon (apache#2199)

   e78854c

   Morsel-driven Parallelism using rayon (apache#2199)

   f43b114

   Morsel-driven Parallelism using rayon (apache#2199)

   0cf6d72
4. alamb

   closed this as completedin #2226on May 4, 2022
5. alamb

   added a commit that references this issue on May 4, 2022

   Introduce new optional scheduler, using Morsel-driven Parallelism + r…

   ...

   dc76ec1
6. JasonLi-cn commented on Sep 30, 2022

   

   JasonLi-cn

   on Sep 30, 2022

   Contributor

   More actions

   1. binary code

   use datafusion::arrow::record_batch::RecordBatch;
   use datafusion::arrow::util::pretty::print_batches;
   use datafusion::error::Result;
   use datafusion::prelude::*;
   use datafusion::scheduler::Scheduler;
   use futures::{StreamExt, TryStreamExt};
   use std::env;
   
   #[tokio::main]
   async fn main() -> Result<()> {
       let name = "test_table";
       let mut args = env::args();
       args.next();
       let table_path = args.next().expect("parquet file");
       let sql = &args.next().expect("sql");
       let using_scheduler = args.next().is_some();
   
       // create local session context
       let config = SessionConfig::new()
           .with_information_schema(true)
           .with_target_partitions(4);
       let context = SessionContext::with_config(config);
   
       // register parquet file with the execution context
       context
           .register_parquet(name, &table_path, ParquetReadOptions::default())
           .await?;
   
       let task = context.task_ctx();
       let query = context.sql(sql).await.unwrap();
       let plan = query.create_physical_plan().await.unwrap();
   
       println!("Start query, using scheduler {}", using_scheduler);
       let now = std::time::Instant::now();
       let results = if using_scheduler {
           let scheduler = Scheduler::new(4);
           let stream = scheduler.schedule(plan, task).unwrap().stream();
           let results: Vec<RecordBatch> = stream.try_collect().await.unwrap();
           results
       } else {
           context.sql(sql).await?.collect().await?
       };
       let elapsed = now.elapsed().as_millis();
       println!("End query, elapsed {} ms", elapsed);
       print_batches(&results)?;
       Ok(())
   }
   
   /// Execute sql
   async fn plan_and_collect(
       context: &SessionContext,
       sql: &str,
   ) -> Result<Vec<RecordBatch>> {
       context.sql(sql).await?.collect().await
   }

   2. test data

   • format: parquet
   • number of files: 4
   • rows: 16405852 * 4 = 65623408
   • number of columns: 6
   • schema: uint32, uint32, uint32, uint32, string, uint32

   3. test result

   SQLs:

   select count(distinct column0) from test_table;
   select * from test_table order by column5 limit 10;

   The performance is similar with and without the Scheduler! Is there a problem with where I use it?

   @tustvold
7. tustvold commented on Sep 30, 2022

   

   tustvold

   on Sep 30, 2022

   ContributorAuthor

   More actions

   Yes that is expected, I've had to park working on this for a bit in favour of some other things. See #2504 for the follow on work
8. JasonLi-cn commented on Sep 30, 2022

   

   JasonLi-cn

   on Sep 30, 2022

   Contributor

   More actions

   Yes that is expected, I've had to park working on this for a bit in favour of some other things. See #2504 for the follow on work

   Ok, thanks!
   By the way, do you know the difference between ClickHouse's Query Execution Pipeline and Datafusion's Execution model(likes vectorized volcano model)? And what are the advantages of ClickHouse?
9. alamb commented on Jun 19, 2024

   

   alamb

   on Jun 19, 2024

   Contributor

   More actions

   Updated description of this ticket to note DF doesn't use morsel driven parallelism, and added link to the paper https://dl.acm.org/doi/10.1145/3626246.3653368

   👍React with 👍1Reacted by Tv