Support Preimage for `date_trunc` by sdf-jkl · Pull Request #22114 · apache/datafusion

sdf-jkl · 2026-05-11T23:25:33Z

Which issue does this PR close?

Closes Optimize the evaluation of DATE_TRUNC(<col>) == <constant>) when pushed down #18319.

Rationale for this change

Enable pruning for predicates with date_trunc on the column side

What changes are included in this PR?

Added preimage implementation for date_trunc
Moved date_to_scalar to a common crate

Are these changes tested?

Yes, unit tests

Are there any user-facing changes?

No

…mage-date_trunc

alamb · 2026-05-12T20:02:43Z

I hope to review this soon

github-actions · 2026-07-12T02:15:36Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

sdf-jkl · 2026-07-20T21:58:57Z

@alamb 😢

sdf-jkl · 2026-07-20T22:42:16Z

@alamb 😢

please nvm, and don't worry about it 🐱

# Conflicts: # datafusion/functions/src/datetime/date_trunc.rs

alamb · 2026-07-22T22:53:57Z

I merged up from main to resolve a conflict

alamb

Thanks @sdf-jkl -- I am sorry that this got left hanging

I started going through it with claude today

One thing if flagged that I haven't had a chance to review is a wrong results bug with wacky timezones -- maybe we can make the code more conservative

I will try and get this reviewed over the next day or two with a fresh set of 👓

🔴 High — wrong results: hour granularity with a named timezone

has_variable_width_in_tz (date_trunc.rs:145) excludes Day and coarser but permits Hour, so the timestamp arm applies a fixed 3600 s step. That holds for every whole-hour DST transition (I confirmed America/New_York fall-back round-trips exactly — _date_trunc_coarse_with_tz disambiguates by offset), but it breaks when a zone's offset shifts by a non-hour amount. Australia/Lord_Howe shifts 30 minutes:

create table lh as select arrow_cast(column1, 'Timestamp(Nanosecond, Some("Australia/Lord_Howe"))') as ts
from (values ('2024-10-05T15:00:00Z'),('2024-10-05T15:30:00Z'),('2024-10-05T15:45:00Z'),
('2024-10-05T16:00:00Z'),('2024-10-05T16:15:00Z'));

-- ground truth (eq on the cast result, so preimage can't fire): 2 rows
SELECT arrow_cast(ts,'Utf8') FROM lh
WHERE arrow_cast(date_trunc('hour', ts),'Utf8') = '2024-10-06T02:30:00+11:00';
-- 2024-10-06T02:30:00+11:00
-- 2024-10-06T02:45:00+11:00

-- with the preimage rewrite: 4 rows
SELECT arrow_cast(ts,'Utf8') FROM lh
WHERE date_trunc('hour', ts) = arrow_cast('2024-10-05T15:30:00Z','Timestamp(Nanosecond, Some("Australia/Lord_Howe"))');
-- 2024-10-06T02:30:00+11:00
-- 2024-10-06T02:45:00+11:00
-- 2024-10-06T03:00:00+11:00 <-- date_trunc gives 03:00, not 02:30
-- 2024-10-06T03:15:00+11:00 <-- same

The spring-forward bucket is 30 minutes wide, not 60. general_date_trunc(15:30) == 15:30 so the alignment check passes, and upper overshoots by 30 min. Since rewrite_with_preimage replaces the predicate rather than refining it, the interval has to be exact.

alamb · 2026-07-22T22:55:58Z

+/// variants encode the bare date; Timestamp variants are anchored at the
+/// date's midnight in the given timezone. Returns `None` for unsupported
+/// target types.
+pub(crate) fn date_to_scalar(


What do you think about making this a method on ScalarValue (like ScalarValue::try_from_date, matching the existing methods like https://docs.rs/datafusion/latest/datafusion/common/enum.ScalarValue.html#method.try_from_string)?

That might make it easier to discover in the future

we can do this as a follow on

codecov-commenter · 2026-07-22T23:20:50Z

Codecov Report

❌ Patch coverage is 69.96700% with 91 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.88%. Comparing base (1da3cf0) to head (d9420a0).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/functions/src/datetime/date_trunc.rs	66.23%	53 Missing and 25 partials ⚠️
datafusion/common/src/scalar/mod.rs	81.42%	6 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #22114      +/-   ##
==========================================
- Coverage   82.89%   82.88%   -0.01%     
==========================================
  Files        1153     1153              
  Lines      457487   457758     +271     
  Branches   457487   457758     +271     
==========================================
+ Hits       379213   379404     +191     
- Misses      55264    55311      +47     
- Partials    23010    23043      +33

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-10-06T01:48:11Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

sdf-jkl · 2026-10-13T15:44:41Z

@alamb do you think there's value in this? If so I can get back to it.

alamb · 2026-10-14T10:36:36Z

I do think there is value @sdf-jkl -- what I haven't been able to do is find enough time to reason through it and convince myself it is correct.

I had been hoping we would have some more engagement from others in the community as well 😢

# Conflicts: # datafusion/functions/src/datetime/date_part.rs # datafusion/functions/src/datetime/date_trunc.rs

alamb · 2026-10-14T11:32:15Z

I got distracted by this PR this morning and spent a while merging it up to main, adding some more tests, and fixing a timezone related issue. Let me know what you think @sdf-jkl -- I am truly sorry it took so long

alamb

Thanks @sdf-jkl -- I went through this PR carefully, and added some more comments, tests and examples. Could you give it a look as well?

mhilton · 2026-10-14T14:51:54Z

+    ///
+    /// Returns `None` if `target_type` is not a date or timestamp type, or if
+    /// midnight on `date` cannot be represented such as when the timezone skips
+    /// midnight that day (for example a daylight saving transition at 00:00).


How important is this last property (that it returns None for date's with no midnight)? The alternative is to return the first timestamp that does exist in the day, probably 01:00. Admittedly that is rather tricky to calculate just now, but once apache/arrow-rs#11535 is available it'd be much simpler.

With this current design there is always a risk of a sudden NULL appearing in a sorted array of dates so operators that use this function cannot be known to preserve the sorting, possibly requiring resorting of sorted data.

alamb · 2026-10-15T20:41:21Z

Thank you for the review @mhilton -- I think I implemented all your comments.

alamb · 2026-10-15T20:41:43Z

    Ok(Arc::new(result))
 }

-fn date_to_scalar(date: NaiveDate, target_type: &DataType) -> Option<ScalarValue> {


moved to method on ScalarValue

alamb · 2026-10-15T20:41:47Z

+    /// `date_trunc` floors a timestamp to the start of the given period, so
+    /// for any aligned literal `L` the preimage is the half-open interval
+    /// `[L, L + 1 granularity)`.
+    ///


Also added some examples

mhilton · 2026-10-16T07:17:20Z

+/// This is local midnight unless the timezone skips midnight that day (for
+/// example a daylight saving transition from 00:00 to 01:00), in which case
+/// it is the first instant after the gap. If midnight occurs twice (a
+/// transition back from 01:00 to 00:00) the earlier instant is returned.


If you want a reference for this it is the convention described in RFC5545§3.3.5

mhilton · 2026-10-16T07:27:50Z

+            // less than a day, so that instant is before the transition), and
+            // use it to convert midnight to the transition instant.
+            let probe = midnight.checked_sub_signed(Duration::days(1))?;
+            let offset_before = tz.offset_from_utc_datetime(&probe).fix();


I'm pretty sure this works, but it's baffling.midnight is a local time so therefore is probe. I think there needs to be a comment saying why it is fine to use it as UTC, I think that's because we just care about some timestamp that is sufficiently before the discontinuity and it doesn't matter exactly what that time is.

I added the comment to explain the reasoning (it was of course written by an LLM but I double checked it and clarified the comment)

alamb · 2026-10-18T11:22:05Z

Thanks again @sdf-jkl and @mhilton

sdf-jkl · 2026-10-18T13:39:27Z

Thanks for pushing this through @alamb @mhilton 🙏

sdf-jkl added 5 commits April 10, 2026 16:27

initial codex pass

a19501c

second pass

be02003

Merge branch 'main' of https://github.com/apache/datafusion into prei…

ba08ebd

…mage-date_trunc

fmt

6927362

moving some unit tests to slt

a0b006a

github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels May 11, 2026

github-actions Bot added the Stale PR has not had any activity for some time label Jul 12, 2026

github-actions Bot closed this Jul 19, 2026

alamb reopened this Jul 22, 2026

Merge remote-tracking branch 'apache/main' into preimage-date_trunc

ba1831d

# Conflicts: # datafusion/functions/src/datetime/date_trunc.rs

alamb changed the title ~~Preimage date trunc~~ Support Preimage for date_trunc Jul 22, 2026

alamb reviewed Jul 22, 2026

View reviewed changes

github-actions Bot removed the Stale PR has not had any activity for some time label Jul 23, 2026

github-actions Bot added the Stale PR has not had any activity for some time label Oct 6, 2026

github-actions Bot closed this Oct 13, 2026

alamb reopened this Oct 14, 2026

alamb added 3 commits October 14, 2026 06:53

Merge remote-tracking branch 'apache/main' into preimage-date_trunc

48fe53b

# Conflicts: # datafusion/functions/src/datetime/date_part.rs # datafusion/functions/src/datetime/date_trunc.rs

Move ScalarValue

8a0886d

Add tests for 30min dst

a9bccbf

alamb mentioned this pull request Oct 14, 2026

unwrap_cast_in_comparison does not unwrap CAST(date_col AS TIMESTAMP) <op> timestamp_literal #26289

Open

alamb approved these changes Oct 14, 2026

View reviewed changes

Comment thread datafusion/functions/src/datetime/date_trunc.rs Outdated

Comment thread datafusion/functions/src/datetime/date_trunc.rs

Comment thread datafusion/functions/src/datetime/date_trunc.rs Outdated

alamb added 2 commits October 14, 2026 07:36

better docs

41df982

simplify

c5b485a

mhilton approved these changes Oct 14, 2026

View reviewed changes

github-actions Bot removed the Stale PR has not had any activity for some time label Oct 15, 2026

mhilton reviewed Oct 15, 2026

View reviewed changes

Comment thread datafusion/common/src/scalar/mod.rs Outdated

alamb added 8 commits October 15, 2026 14:08

Merge remote-tracking branch 'apache/main' into preimage-date_trunc

f18b416

descibe why nanoseconds can fail

8105f8d

more comment

9094b78

improve ScalarValue::try_as_timestamp

3b1775f

add spec link

6119b33

combine match

e0bfbec

change semantics to first instant of date

f6d67b4

improve comments

75274b2

alamb mentioned this pull request Oct 15, 2026

Casting Timestamp to Date64 keeps the time of day, producing invalid Date64 values apache/arrow-rs#11592

Closed

remove replicative rust tests

abfc8d3

alamb reviewed Oct 15, 2026

View reviewed changes

mhilton approved these changes Oct 16, 2026

View reviewed changes

alamb added 3 commits October 16, 2026 16:57

document more about transition times

3db471f

Merge remote-tracking branch 'apache/main' into preimage-date_trunc

1ebb512

clarify

d9420a0

alamb added this pull request to the merge queue Oct 18, 2026

Merged via the queue into apache:main with commit 0ee3978 Oct 18, 2026
42 checks passed

github-actions Bot added the v56.0.0 label Oct 18, 2026

sdf-jkl deleted the preimage-date_trunc branch October 18, 2026 13:38

Conversation

sdf-jkl commented May 11, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented May 12, 2026

Uh oh!

github-actions Bot commented Jul 12, 2026

Uh oh!

sdf-jkl commented Jul 20, 2026

Uh oh!

sdf-jkl commented Jul 20, 2026

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

sdf-jkl commented Oct 13, 2026

Uh oh!

alamb commented Oct 14, 2026

Uh oh!

alamb commented Oct 14, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Oct 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Oct 18, 2026

Uh oh!

Uh oh!

sdf-jkl commented Oct 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Jul 22, 2026 •

edited

Loading