Repository navigation
doc: Add Quickstart Comet doc section #125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,3 +58,49 @@ Linux, Apple OSX (Intel and M1) | |
| - Apache Spark 3.2, 3.3, or 3.4 | ||
| - JDK 8 and up | ||
| - GLIBC 2.17 (Centos 7) and up | ||
|
|
||
| ## Getting started | ||
|
|
||
| Make sure the requirements above are met and software installed on your machine | ||
|
|
||
| ### Clone repo | ||
| ```commandline | ||
| git clone https://github.com/apache/arrow-datafusion-comet.git | ||
| ``` | ||
|
|
||
| ### Specify the Spark version and build the Comet | ||
| Spark 3.4 used for the example. | ||
| ``` | ||
| cd arrow-datafusion-comet | ||
| make release PROFILES="-Pspark-3.4" | ||
| ``` | ||
|
|
||
| ### Run Spark with Comet enabled | ||
| Make sure `SPARK_HOME` points to the same Spark version as Comet has built for. | ||
|
|
||
| ``` | ||
| $SPARK_HOME/bin/spark-shell --jars spark/target/comet-spark-spark3.4_2.12-0.1.0-SNAPSHOT.jar \ | ||
| --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \ | ||
| --conf spark.comet.enabled=true \ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we need a separate section to guide through shuffle related settings, but can be done later. |
||
| --conf spark.comet.exec.enabled=true \ | ||
| --conf spark.comet.exec.all.enabled=true | ||
| ``` | ||
|
|
||
| ### Verify Comet enabled for Spark SQL query | ||
|
|
||
| Create a test Parquet source | ||
| ```scala | ||
| scala> (0 until 10).toDF("a").write.mode("overwrite").parquet("/tmp/test") | ||
| ``` | ||
|
|
||
| Query the data from the test source and check: | ||
| - INFO message shows the native Comet library has been initialized. | ||
| - The query plan reflects Comet operators being used for this query instead of Spark ones | ||
| ```scala | ||
| scala> spark.read.parquet("/tmp/test").createOrReplaceTempView("t1"); spark.sql("select * from t1 where a > 5").explain | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It should be nicer to read if newline is added.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll fix it. |
||
| INFO src/lib.rs: Comet native library initialized | ||
| == Physical Plan == | ||
| *(1) ColumnarToRow | ||
| +- CometFilter [a#14], (isnotnull(a#14) AND (a#14 > 5)) | ||
| +- CometScan parquet [a#14] Batched: true, DataFilters: [isnotnull(a#14), (a#14 > 5)], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test], PartitionFilters: [], PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema: struct<a:int> | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. And the explained plan string too. Right now it is a long string which is hard to read.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'll break it into some newlines for easier reading
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Filed #128 |
||
| ``` | ||
Uh oh!
There was an error while loading. Please reload this page.