[Feature] Support Spark expression: datetime_sub #3134

What is the problem the feature request solves?

Note: This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.

Comet does not currently support the Spark datetime_sub function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.

DatetimeSub is a runtime-replaceable expression that subtracts an interval from a timestamp or date value. It serves primarily as a SQL presentation layer for datetime subtraction operations, providing a clean string representation while delegating actual computation to its replacement expression.

Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.

Describe the potential solution

Spark Specification

Syntax:

datetime_column - INTERVAL value unit
timestamp_column - INTERVAL '1' DAY

Arguments:

Return Type: The return type depends on the replacement expression, but typically returns:

• TimestampType when subtracting from timestamp values
• DateType when subtracting from date values

Supported Data Types:

• start: TimestampType, DateType
• interval: CalendarIntervalType, interval literals
• Input types are validated by the replacement expression during analysis

Edge Cases:

• Null handling: Behavior depends on the replacement expression, typically follows SQL null semantics
• Invalid intervals: Runtime errors may occur for malformed interval expressions
• Overflow scenarios: Large interval subtractions may cause timestamp overflow
• Type mismatches: Analysis phase will validate compatible types between start and interval

Examples:

-- Subtract 1 day from current timestamp
SELECT current_timestamp() - INTERVAL '1' DAY;

-- Subtract multiple units
SELECT date_col - INTERVAL '1 year 2 months 3 days' FROM table;

// DataFrame API usage
import org.apache.spark.sql.functions._

df.select(col("timestamp_col") - expr("INTERVAL '1' HOUR"))
df.withColumn("yesterday", col("date_col") - expr("INTERVAL '1' DAY"))

Implementation Approach

See the Comet guide on adding new expressions for detailed instructions.

1. Scala Serde: Add expression handler in spark/src/main/scala/org/apache/comet/serde/
2. Register: Add to appropriate map in QueryPlanSerde.scala
3. Protobuf: Add message type in native/proto/src/proto/expr.proto if needed
4. Rust: Implement in native/spark-expr/src/ (check if DataFusion has built-in support first)

Additional context

Difficulty: Medium
Spark Expression Class: org.apache.spark.sql.catalyst.expressions.DatetimeSub

Related:

• DatetimeAdd - Addition counterpart for datetime arithmetic
• TimeAdd - Time-specific addition operations
• DateAdd - Date-specific addition operations
• RuntimeReplaceable - Base trait for expressions with compile-time replacements

This issue was auto-generated from Spark reference documentation.