Repository navigation

feat: Add `datafusion-spark` crate - #14392

Closed

andygrove wants to merge 69 commits into

apache:mainfrom

andygrove:add-spark-expr-crate

Closed

feat: Add `datafusion-spark` crate#14392
andygrove wants to merge 69 commits into
apache:mainfrom
andygrove:add-spark-expr-crate

Commits on Jul 10, 2024
- feat: Create new `datafusion-comet-spark-expr` crate containing Spark-compatible DataFusion expressions (#638)
  andygrove
  authored
  33f3e9c
Commits on Jul 11, 2024
- feat: Move `IfExpr` to `spark-expr` crate (#653)
  andygrove
  authored
  96a2f41
Commits on Jul 12, 2024
Commits on Jul 15, 2024
- chore: Move temporal kernels and expressions to spark-expr crate (#660)
  andygrove
  authored
  d510649
- fix: Optimize some functions to rewrite dictionary-encoded strings (#627)
  vaibhawvipul
  authored
  46e8bf2
Commits on Jul 16, 2024
- Change suffix on some expressions from Exec to Expr (#673)
  andygrove
  authored
  2179331
Commits on Jul 20, 2024
- chore: Disable abs and signum because they return incorrect results (#695)
  andygrove
  authored
  01e21a9
Commits on Jul 21, 2024
- chore: Make rust clippy happy (#701)
  Xuanwo
  authored
  01362b5
Commits on Jul 24, 2024
- perf: Optimize IfExpr by delegating to CaseExpr (#681)
  andygrove
  authored
  e2d838e
Commits on Jul 27, 2024
- chore: make Cast's logic reusable for other projects (#716)
  Blizzara
  authored
  5dcf713
Commits on Jul 28, 2024
- chore: move scalar_funcs into spark-expr (#712)
  Blizzara
  authored
  2a4dc7b
Commits on Jul 31, 2024
- chore: Add criterion benchmark for decimal_div (#743)
  andygrove
  authored
  0a00325
Commits on Aug 1, 2024
- Add support for time-zone, 3 & 5 digit years: Cast from string to timestamp (#704)
  akhilss99
  authored
  abfce13
- fix: Remove castting on decimals with a small precision to decimal256 (#741)
  kazuyukitanimura
  authored
  1a02d58
Commits on Aug 2, 2024
- feat: Implement basic version of RLIKE (#734)
  andygrove
  authored
  33f1ce9
Commits on Aug 3, 2024
- feat: Add GetStructField expression (#731)
  Kimahriman
  authored
  607ee7d
Commits on Aug 8, 2024
- fix: Optimize read_side_padding (#772)
  kazuyukitanimura
  authored
  e3709ea
Commits on Aug 9, 2024
- feat: Optimze CreateNamedStruct preserve dictionaries (#789)
  eejbyfeldt
  authored
  d2853ef
Commits on Aug 21, 2024
- feat: Enable `clippy::clone_on_ref_ptr` on `proto` and `spark_exprs` crates (#859)
  comphead
  authored
  5a37afc
Commits on Aug 28, 2024
- feat: Implement basic version of string to float/double/decimal (#870)
  andygrove
  authored
  3cf7c4f
- feat: Implement to_json for subset of types (#805)
  
  andygrove
  and
  eejbyfeldt
  authored
  9d86fa9
Commits on Aug 30, 2024
- feat: Support sort merge join with a join condition (#553)
  viirya
  authored
  d9eecde
Commits on Sep 5, 2024
- feat: Array element extraction (#899)
  Kimahriman
  authored
  74097da
- chore: Upgrade to latest DataFusion revision (#909)
  andygrove
  authored
  b5763f6
- build: fix build (#917)
  andygrove
  authored
  c7ed2eb
Commits on Sep 16, 2024
- feat: date_add and date_sub functions (#910)
  mbutrovich
  authored
  c7ec300
Commits on Oct 7, 2024
- feat: Support `GetArrayStructFields` expression (#993)
  Kimahriman
  authored
  ff41f1b
Commits on Oct 14, 2024
- chore: Bump arrow-rs to 53.1.0 and datafusion (#1001)
  kazuyukitanimura
  authored
  e1e5483
Commits on Oct 30, 2024
- chore: Use twox-hash 2.0 xxhash64 oneshot api instead of custom implementation (#1041)
  NoeB
  authored
  2fea49f
Commits on Nov 5, 2024
- chore: Upgrade to DataFusion 43.0.0-rc1 (#1057)
  andygrove
  authored
  118701a
Commits on Nov 8, 2024
- feat: Implement CAST from struct to string (#1066)
  andygrove
  authored
  850cde4
Commits on Nov 11, 2024
- feat: Implement CAST between struct types (#1074)
  andygrove
  authored
  b534608
Commits on Nov 19, 2024
- fix: Unsigned type related bugs (#1095)
  kazuyukitanimura
  authored
  e07a64b
Commits on Nov 22, 2024
- feat: support array_insert (#1073)
  
  SemyonSinchenko
  and
  andygrove
  authored
  97b750e
Commits on Dec 6, 2024
- chore: Refactor cast to use SparkCastOptions param (#1146)
  andygrove
  authored
  7fff547
Commits on Dec 9, 2024
- chore: Move more expressions from core crate to spark-expr crate (#1152)
  andygrove
  authored
  df2350f
Commits on Dec 12, 2024
- chore: Move string kernels and expressions to spark-expr crate (#1164)
  andygrove
  authored
  3859724
- chore: Move remaining expressions to spark-expr crate + some minor refactoring (#1165)
  andygrove
  authored
  f69148d
Commits on Dec 17, 2024
- feat: Add Spark-compatible implementation of SchemaAdapterFactory (#1169)
  andygrove
  authored
  7bd99f6
Commits on Dec 19, 2024
- fix: stddev_pop should not directly return 0.0 when count is 1.0 (#1184)
  viirya
  authored
  67038db
Commits on Dec 28, 2024
- chore: Upgrade to DataFusion 44.0.0-rc2 (#1154)
  
  andygrove
  and
  alamb
  authored
  cf44902
Commits on Jan 6, 2025
- extract struct expressions to folders based on spark grouping (#1216)
  rluvaton
  authored
  4af3a2f
- chore: extract static invoke expressions to folders based on spark grouping (#1217)
  
  rluvaton
  and
  andygrove
  authored
  11421f7
Commits on Jan 7, 2025
- chore: extract agg_funcs expressions to folders based on spark grouping (#1224)
  rluvaton
  authored
  37b7acb
- extract datetime_funcs expressions to folders based on spark grouping (#1222)
  
  rluvaton
  and
  andygrove
  authored
  ac22924
Commits on Jan 8, 2025
Commits on Jan 9, 2025
- chore: extract hash_funcs expressions to folders based on spark grouping (#1221)
  
  rluvaton
  and
  andygrove
  authored
  31a725f
Commits on Jan 18, 2025
- chore: extract conversion_funcs, conditional_funcs, bitwise_funcs and array_funcs expressions to folders based on spark grouping (#1223)
  rluvaton
  authored
  2cd57b5
Commits on Jan 19, 2025
- fix: partially fix consistency issue of hash functions with decimal input (#1295)
  wForget
  authored
  ba08511
Commits on Jan 20, 2025
- chore: extract math_funcs expressions to folders based on spark grouping (#1219)
  rluvaton
  authored
  219a133
Commits on Jan 21, 2025
- chore: merge comet-parquet-exec branch into main (#1318)
  andygrove
  authored
  bbb8548
Commits on Jan 22, 2025
- fix merge conflicts (#1320)
  andygrove
  authored
  1ba1e2b
Commits on Jan 23, 2025
- chore: Fix merge conflicts from merging comet-parquet-exec into main (#1323)
  mbutrovich
  authored
  d9fe1b3
Commits on Jan 30, 2025
- chore: Prepare for DataFusion 45 (bump to DataFusion rev 5592834 + Arrow 54.0.0) (#1332)
  andygrove
  authored
  35821f1
Commits on Jan 31, 2025
- Add spark-expr crate
  andygrove
  committed
  fc41754
- move benches
  andygrove
  committed
  53aec1a
- fix Cargo.toml
  andygrove
  committed
  1c0901a
- fix header format
  andygrove
  committed
  c404172
- add to workspace
  andygrove
  committed
  024912a
- prettier
  andygrove
  committed
  071268d
- fmt
  andygrove
  committed
  f563cb7
- fix merge conflict in README
  andygrove
  committed
  af2ec26
- fix cargo doc failures
  andygrove
  committed
  d7eaf68
- taplo fmt
  andygrove
  committed
  96f2136
- rename functions-spark to spark
  andygrove
  committed
  77e7831

Commits

Commits on Jul 10, 2024

Commits on Jul 11, 2024

Commits on Jul 12, 2024

Commits on Jul 15, 2024

Commits on Jul 16, 2024

Commits on Jul 20, 2024

Commits on Jul 21, 2024

Commits on Jul 24, 2024

Commits on Jul 27, 2024

Commits on Jul 28, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 2, 2024

Commits on Aug 3, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 21, 2024

Commits on Aug 28, 2024

Commits on Aug 30, 2024

Commits on Sep 5, 2024

Commits on Sep 16, 2024

Commits on Oct 7, 2024

Commits on Oct 14, 2024

Commits on Oct 30, 2024

Commits on Nov 5, 2024

Commits on Nov 8, 2024

Commits on Nov 11, 2024

Commits on Nov 19, 2024

Commits on Nov 22, 2024

Commits on Dec 6, 2024

Commits on Dec 9, 2024

Commits on Dec 12, 2024

Commits on Dec 17, 2024

Commits on Dec 19, 2024

Commits on Dec 28, 2024

Commits on Jan 6, 2025

Commits on Jan 7, 2025

Commits on Jan 8, 2025

Commits on Jan 9, 2025

Commits on Jan 18, 2025

Commits on Jan 19, 2025

Commits on Jan 20, 2025

Commits on Jan 21, 2025

Commits on Jan 22, 2025

Commits on Jan 23, 2025

Commits on Jan 30, 2025

Commits on Jan 31, 2025