From 31f8d2d2e61846511c644602315acdd154190fac Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 15 Jan 2026 12:48:33 -0500 Subject: [PATCH 1/3] Docs: add additional links to blog posts --- .../library-user-guide/extending-sql.md | 5 ++++ .../functions/adding-udfs.md | 4 +++ .../library-user-guide/query-optimizer.md | 12 ++++++++ .../user-guide/concepts-readings-events.md | 28 +++++++++++++++++++ docs/source/user-guide/sql/data_types.md | 5 ++++ 5 files changed, 54 insertions(+) diff --git a/docs/source/library-user-guide/extending-sql.md b/docs/source/library-user-guide/extending-sql.md index 409a0fb89a321..524948b444a2d 100644 --- a/docs/source/library-user-guide/extending-sql.md +++ b/docs/source/library-user-guide/extending-sql.md @@ -27,6 +27,11 @@ need to: - Add custom data types not natively supported - Implement SQL constructs like `TABLESAMPLE`, `PIVOT`/`UNPIVOT`, or `MATCH_RECOGNIZE` +You can read more about this topic in the [Extending SQL in DataFusion: from ->> +to TABLESAMPLE] blog. + +[Extending SQL in DataFusion: from ->> to TABLESAMPLE]: https://datafusion.apache.org/blog/2026/01/12/extending-sql + ## Architecture Overview When DataFusion processes a SQL query, it goes through these stages: diff --git a/docs/source/library-user-guide/functions/adding-udfs.md b/docs/source/library-user-guide/functions/adding-udfs.md index 5d033ae3f9e97..a645c12827649 100644 --- a/docs/source/library-user-guide/functions/adding-udfs.md +++ b/docs/source/library-user-guide/functions/adding-udfs.md @@ -684,6 +684,10 @@ No function matches the given name and argument types substr(Utf8). Scalar UDFs are functions that take a row of data and return a single value. Window UDFs are similar, but they also have access to the rows around them. Access to the proximal rows is helpful, but adds some complexity to the implementation. +For background and other considerations, see the +[User defined Window Functions in DataFusion] blog. + +[User defined Window Functions in DataFusion]: https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions For example, we will declare a user defined window function that computes a moving average. ```rust diff --git a/docs/source/library-user-guide/query-optimizer.md b/docs/source/library-user-guide/query-optimizer.md index 8ed6593d56203..36fc9cd33c11b 100644 --- a/docs/source/library-user-guide/query-optimizer.md +++ b/docs/source/library-user-guide/query-optimizer.md @@ -28,9 +28,21 @@ This crate is a submodule of DataFusion that provides a query optimizer for logi contains an extensive set of [`OptimizerRule`]s and [`PhysicalOptimizerRules`] that may rewrite the plan and/or its expressions so they execute more quickly while still computing the same result. +For a deeper background on optimizer architecture and rule types and predicates, see +[Optimizing SQL (and DataFrames) in DataFusion, Part 1], [Part 2], +[Using Ordering for Better Plans in Apache DataFusion], and +[Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries]. + [`optimizerrule`]: https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html [`physicaloptimizerrules`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/trait.PhysicalOptimizerRule.html +[Optimizing SQL (and DataFrames) in DataFusion, Part 1]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one +[Part 2]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two +[Using Ordering for Better Plans in Apache DataFusion]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis +[Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters + + + ## Running the Optimizer The following code demonstrates the basic flow of creating the optimizer with a default set of optimization rules diff --git a/docs/source/user-guide/concepts-readings-events.md b/docs/source/user-guide/concepts-readings-events.md index ad444ef91c474..5f6f30c1e719d 100644 --- a/docs/source/user-guide/concepts-readings-events.md +++ b/docs/source/user-guide/concepts-readings-events.md @@ -37,6 +37,34 @@ This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find +- **2026-01-12** [Blog: Extending SQL in DataFusion: from ->> to TABLESAMPLE](https://datafusion.apache.org/blog/2026/01/12/extending-sql) + +- **2025-12-15** [Blog: Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution](https://datafusion.apache.org/blog/2025/12/15/avoid-consecutive-repartitions) + +- **2025-09-21** [Blog: Implementing User Defined Types and Custom Metadata in DataFusion](https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata) + +- **2025-09-10** [Blog: Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries](https://datafusion.apache.org/blog/2025/09/10/dynamic-filters) + +- **2025-08-15** [Blog: Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet](https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes) + +- **2025-07-14** [Blog: Embedding User-Defined Indexes in Apache Parquet Files](https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes) + +- **2025-06-30** [Blog: Using Rust async for Query Execution and Cancelling Long-Running Queries](https://datafusion.apache.org/blog/2025/06/30/cancellation) + +- **2025-06-15** [Blog: Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview](https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one) + +- **2025-06-15** [Blog: Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion](https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two) + +- **2025-04-19** [Blog: User defined Window Functions in DataFusion](https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions) + +- **2025-04-10** [Blog: tpchgen-rs World's fastest open source TPC-H data generator, written in Rust](https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator) + +- **2025-03-11** [Blog: Using Ordering for Better Plans in Apache DataFusion](https://datafusion.apache.org/blog/2025/03/11/ordering-analysis) + +- **2024-05-07** [Blog: Announcing Apache Arrow DataFusion is now Apache DataFusion](https://datafusion.apache.org/blog/2024/05/07/datafusion-tlp) + +- **2024-03-06** [Blog: Announcing Apache Arrow DataFusion Comet](https://datafusion.apache.org/blog/2024/03/06/comet-donation) + - **2025-03-21** [Blog: Efficient Filter Pushdown in Parquet](https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown/) - **2025-03-20** [Blog: Parquet Pruning in DataFusion: Read Only What Matters](https://datafusion.apache.org/blog/2025/03/20/parquet-pruning/) diff --git a/docs/source/user-guide/sql/data_types.md b/docs/source/user-guide/sql/data_types.md index 02edb6371ce3e..8bb054c6f6a1f 100644 --- a/docs/source/user-guide/sql/data_types.md +++ b/docs/source/user-guide/sql/data_types.md @@ -25,6 +25,11 @@ execution. The SQL types from are mapped to [Arrow data types](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html) according to the following table. This mapping occurs when defining the schema in a `CREATE EXTERNAL TABLE` command or when performing a SQL `CAST` operation. +For background on extension types and custom metadata, see the +[Implementing User Defined Types and Custom Metadata in DataFusion] blog. + +[Implementing User Defined Types and Custom Metadata in DataFusion]: https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata + You can see the corresponding Arrow type for any SQL expression using the `arrow_typeof` function. For example: From 46f1c6445b9324bc5ce98622b65abb6210e2e4c0 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 15 Jan 2026 12:59:31 -0500 Subject: [PATCH 2/3] prettier --- docs/source/library-user-guide/extending-sql.md | 2 +- .../library-user-guide/functions/adding-udfs.md | 7 ++++--- docs/source/library-user-guide/query-optimizer.md | 15 ++++++--------- docs/source/user-guide/sql/data_types.md | 2 +- 4 files changed, 12 insertions(+), 14 deletions(-) diff --git a/docs/source/library-user-guide/extending-sql.md b/docs/source/library-user-guide/extending-sql.md index 524948b444a2d..b4d2519dd6264 100644 --- a/docs/source/library-user-guide/extending-sql.md +++ b/docs/source/library-user-guide/extending-sql.md @@ -30,7 +30,7 @@ need to: You can read more about this topic in the [Extending SQL in DataFusion: from ->> to TABLESAMPLE] blog. -[Extending SQL in DataFusion: from ->> to TABLESAMPLE]: https://datafusion.apache.org/blog/2026/01/12/extending-sql +[extending sql in datafusion: from ->> to tablesample]: https://datafusion.apache.org/blog/2026/01/12/extending-sql ## Architecture Overview diff --git a/docs/source/library-user-guide/functions/adding-udfs.md b/docs/source/library-user-guide/functions/adding-udfs.md index a645c12827649..419ec6c943b88 100644 --- a/docs/source/library-user-guide/functions/adding-udfs.md +++ b/docs/source/library-user-guide/functions/adding-udfs.md @@ -684,10 +684,11 @@ No function matches the given name and argument types substr(Utf8). Scalar UDFs are functions that take a row of data and return a single value. Window UDFs are similar, but they also have access to the rows around them. Access to the proximal rows is helpful, but adds some complexity to the implementation. -For background and other considerations, see the -[User defined Window Functions in DataFusion] blog. +For background and other considerations, see the +[User defined Window Functions in DataFusion] blog. + +[user defined window functions in datafusion]: https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions -[User defined Window Functions in DataFusion]: https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions For example, we will declare a user defined window function that computes a moving average. ```rust diff --git a/docs/source/library-user-guide/query-optimizer.md b/docs/source/library-user-guide/query-optimizer.md index 36fc9cd33c11b..3668fc7084da7 100644 --- a/docs/source/library-user-guide/query-optimizer.md +++ b/docs/source/library-user-guide/query-optimizer.md @@ -28,20 +28,17 @@ This crate is a submodule of DataFusion that provides a query optimizer for logi contains an extensive set of [`OptimizerRule`]s and [`PhysicalOptimizerRules`] that may rewrite the plan and/or its expressions so they execute more quickly while still computing the same result. -For a deeper background on optimizer architecture and rule types and predicates, see +For a deeper background on optimizer architecture and rule types and predicates, see [Optimizing SQL (and DataFrames) in DataFusion, Part 1], [Part 2], -[Using Ordering for Better Plans in Apache DataFusion], and +[Using Ordering for Better Plans in Apache DataFusion], and [Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries]. [`optimizerrule`]: https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html [`physicaloptimizerrules`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/trait.PhysicalOptimizerRule.html - -[Optimizing SQL (and DataFrames) in DataFusion, Part 1]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one -[Part 2]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two -[Using Ordering for Better Plans in Apache DataFusion]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis -[Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters - - +[optimizing sql (and dataframes) in datafusion, part 1]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one +[part 2]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two +[using ordering for better plans in apache datafusion]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis +[dynamic filters: passing information between operators during execution for 25x faster queries]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters ## Running the Optimizer diff --git a/docs/source/user-guide/sql/data_types.md b/docs/source/user-guide/sql/data_types.md index 8bb054c6f6a1f..4841098dd534b 100644 --- a/docs/source/user-guide/sql/data_types.md +++ b/docs/source/user-guide/sql/data_types.md @@ -28,7 +28,7 @@ This mapping occurs when defining the schema in a `CREATE EXTERNAL TABLE` comman For background on extension types and custom metadata, see the [Implementing User Defined Types and Custom Metadata in DataFusion] blog. -[Implementing User Defined Types and Custom Metadata in DataFusion]: https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata +[implementing user defined types and custom metadata in datafusion]: https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata You can see the corresponding Arrow type for any SQL expression using the `arrow_typeof` function. For example: From efd68f09913fcde6141267f419f4f37cccd89264 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 15 Jan 2026 13:00:11 -0500 Subject: [PATCH 3/3] tweak --- docs/source/library-user-guide/functions/adding-udfs.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/source/library-user-guide/functions/adding-udfs.md b/docs/source/library-user-guide/functions/adding-udfs.md index 419ec6c943b88..c707495737f46 100644 --- a/docs/source/library-user-guide/functions/adding-udfs.md +++ b/docs/source/library-user-guide/functions/adding-udfs.md @@ -684,8 +684,7 @@ No function matches the given name and argument types substr(Utf8). Scalar UDFs are functions that take a row of data and return a single value. Window UDFs are similar, but they also have access to the rows around them. Access to the proximal rows is helpful, but adds some complexity to the implementation. -For background and other considerations, see the -[User defined Window Functions in DataFusion] blog. +For background and other considerations, see the [User defined Window Functions in DataFusion] blog. [user defined window functions in datafusion]: https://datafusion.apache.org/blog/2025/04/19/user-defined-window-functions