[Proposal] Decouple logical from physical types

Abstract

Logical types are an abstract representation of data that emphasises structure without regard for physical implementation, while physical types are tangible representations that describe how data will be stored, accessed, and retrieved.

Currently the type model in DataFusion, both in logical and physical plans, relies purely on arrow’s DataType enum. But whilst this choice makes sense its physical execution engine (DataTypes map 1:1 with the physical array representation, defining implicit rules for storage, access, and retrieval), it has flaws when dealing with its logical plans. This is due to the fact that some logically equivalent array types in the Arrow format have very different DataTypes – for example a logical string array can be represented in the Arrow array of DataType Utf8, Dictionary(Utf8), RunEndEncoded(Utf8), and StringView (without mentioning the different indexes types that can be specified for dictionaries and REE arrays).

This proposal evaluates possible solutions for decoupling the notion of types in DataFusion’s logical plans with DataTypes, evaluating their impact on DataFusion itself and on downstream crates.

Goals

Introduce a subset of DataTypes to be used as logical types
Define the boundaries of logical types and physical types in the planning and execution path
Define a solution to keep track of a column’s logical and physical type
Introduce a new logical ScalarValue
Define how to specify materialisation options for physical types
Evaluate the impact on downstream crates

Proposal

Defining a logical type

To define the list of logical types we must first take a look at the physical representation of the engine: the Arrow columnar format. DataTypes are the physical types of the DataFusion engine and they define storage and access pattern for buffers in the Arrow format.

Looking at a list of the possible DataTypes it's clear that while some map 1:1 with their logical representation other also specify information about the encoding (e.g. Large*, FixedSize*, Dictionary , RunEndEncoded...). The latter must be consolidate into what they represent, discarding the encoding information and, in general, types that can store different ranges of values should be different logical types. (ref).

What follows is a list of DataTypes and how would they map to their respective logical type following the rules above:

DataType (aka physical type)	Logical type	Note
`Null`	`Null`
`Boolean`	`Boolean`
`Int8`	`Int8`
`Int16`	`Int16`
`Int32`	`Int32`
`Int64`	`Int64`
`UInt8`	`UInt8`
`UInt16`	`Uint16`
`UInt32`	`UInt32`
`UInt64`	`UInt64`
`Float16`	`Float16`
`Float32`	`Float32`
`Float64`	`Float64`
`Timestamp(unit, tz)`	`Timestamp(unit, tz)`
`Date32`	`Date`
`Date64`	`Date`	`Date64` doesn't actually provide more precision. (docs)
`Time32(unit)`	`Time32(unit)`
`Time64(unit)`	`Time64(unit)`
`Duration(unit)`	`Duration(uint)`
`Interval(unit)`	`Interval(unit)`
`Binary`	`Binary`
`FixedSizeBinary(size)`	`Binary`
`LargeBinary`	`Binary`
`BinaryView`	`Binary`
`Utf8`	`Utf8`
`LargeUtf8`	`Utf8`
`Utf8View`	`Utf8`
`List(field)`	`List(field)`
`ListView(field)`	`List(field)`
`FixedSizeList(field, size)`	`List(field)`
`LargeList(field)`	`List(field)`
`LargeListView(field)`	`List(field)`
`Struct(fields)`	`Struct(fields)`
`Union(fields, mode)`	`Union(fields)`
`Dictionary(index_type, data_type)`	underlying `data_type`, converted to logical type
`Decimal128(precision, scale)`	`Decimal128(precision, scale)`
`Decimal256(precision, scale)`	`Decimal256(precision, scale)`
`Map(fields, sorted)`	`Map(fields, sorted)`
`RunEndEncoded(run_ends_type, data_type)`	underlying `data_type`, converted to logical type

User defined types

User defined physical types

The Arrow columnar format provides guidelines to define Extension types though the composition of native DataTypes and custom metadata in fields. Since this proposal already includes a mapping from DataType to logical type we could extend it to support user defined types (through extension types) which would map to a known logical type.

For example an extension type with the DataType of List(UInt8) and a custom metadata field {'ARROW:extension:name': 'myorg.mystring'} could have a logical type of Utf8.

User defined logical types

Arrow extension types can also be used to extend the list of supported logical types. An additional logical type called Extension could be introduced. This extension type would contain a structure detailing its logical type and the extension type metadata.

Boundaries of logical and physical types

In plans and expressions

As the prefix suggests, logical types should be used exclusively in logical plans (LogicalPlan and Expr) while physical types should be used exclusively in physical plans (ExecutionPlan and PhysicalExpr). This would enable logical plans to be purely logical, without worrying about underlying encodings.

Expr in logical plans would need to represent their resulting value as logical types through the trait method ExprSchemable::get_type, which would need to return a logical type instead.

In functions

ScalarUDF, WindowUDF, and AggregateUDF all define their Signatures through the use of DataTypes. Function arguments are currently validated against signatures through type coercion during logical planning. With logical types Signatures would be expressed without the need to specify the underlying encoding. This would simplify the type coercion rules, removing the need of traversing dictionaries and handling different containers and focusing instead on explicit logical rules (e.g. all logical types can be coerced to Utf8).

During execution the function receives a slice of ColumnarValue that is guaranteed to match the signature. Being strictly a physical operation, the function will have to deal with physical types. ColumnarValue enum could be extended so that functions could choose to provide their own optimised implementation for a subset of physical types and then fall back to a generic implementation that materialises the argument to known physical type. This would potentially allow native functions to support user defined physical types that map to known logical types.

In substrait

The datafusion_substrait crate provides helper functions to enable interoperability between substrait plans and datafusion's plan. While some effort has been made to support converting from / to DataTypes via type_variation_reference (example here), dictionaries and not supported as both literal types and cast types, leading to potential errors when trying to encode a valid LogicalPlan) into a substrait plan. The usage of logical types would enable a more seamless transition between DataFusion's native logical plan and substrait.

Keeping track of the physical type

While logical types simplify the list of possible types that can be handled during logical planning, the relation to their underlying physical representation needs to be accounted for when transforming the logical plan into nested ExecutionPlan and PhysicalExpr which will dictate how will the query execute.

This proposal introduces a new trait that represents the link between a logical type and its underlying physical representation:

pub enum LogicalType {/*...*/}

pub type TypeRelationRef = Arc<dyn TypeRelation + Send + Sync>;

pub trait TypeRelation: std::fmt::Debug {
    fn logical(&self) -> &LogicalType;
    fn physical(&self) -> &DataType;
    // ...
}

#[derive(Clone, Debug)]
pub struct NativeType {
    logical: LogicalType,
    physical: DataType,
}

impl TypeRelation for NativeType {/*...*/}
impl From<DataType> for NativeType {/*...*/}
impl From<DataType> for LogicalType {/*...*/}


#[derive(Clone, Debug)]
pub struct LogicalPhysicalType(TypeRelationRef);

impl TypeRelation for LogicalPhysicalType {/*...*/}

While NativeType would be primarily used for standard DataTypes and their logical relation, TypeRelation is defined to provide support for used defined physical types.

What follows is an exploration of the areas in which LogicalPhysicalType would need to get introduced:

A new type of Schema

To support the creation of LogicalPhysicalType a new schema must be introduced, which can be consumed as either a logical schema or used to access the underlying physical representation. Currently DFSchema is used throughout DataFusion as a thin wrapper for Arrow's native Schema in order to qualify fields originating from potentially different tables. This proposal suggest to decouple the DFSchema from its underlying Schema and instead adopt a new Schema-compatible structure (LogicalPhysicalSchema) but with DataTypes replaced by LogicalPhysicalType. This would also mean the introduction of new Field-compatible structure (LogicalPhysicalField) which also supports LogicalPhysicalType instead of Arrow's native Field DataType.

DFSchema would be used by DataFusion's planning and execution engine to derive either logical or physical type information of each field. It should retain the current interoperability with Schema (and additionally the new LogicalPhysicalSchema) allowing easy Into & From conversion.

Type sources

Types in plans sourced through Arrow's native Schema returned by implementations of TableSource / TableProvider , variables DataTypes returned by VarProvider , and ScalarValue. To allow definition of custom LogicalPhysicalType these type sources should be edited to return LogicalPhysicalSchema / LogicalPhysicalType.

Tables

For tables a non-breaking way of editing the trait to support LogicalPhysicalSchema could be:

Declare a new trait method logical_physical_schema() -> LogicalPhysicalSchema, this method's default implementation calls the schema() and converts it to LogicalPhysicalSchema without introducing any custom LogicalPhysicalType. Implementers are free to override this method and add custom LogicalPhysicalType.
Declare the existing schema() method to return impl Into<LogicalPhysicalSchema>.

Open question: Should we really introduce a new schema type or should we reuse DFSchema? The qualifiers for fields in DFSchema should not be specified by the Tables themselves.

VarProvider

VarProvider needs to be edited in order to return a LogicalPhysicalType when getting the type of the variable, while the actual variable can very well remain a ScalarValue.

ScalarValue

ScalarValue should be wrapped in order to have a way of retrieving both its logical and its underlying physical type. When reasoning about logical plans it should be treated as its logical type while its physical properties should be accessed exclusively by the physical plan.

Open question: Should ScalarValue split into a LogicalScalarValue and a PhysicalScalarValue? Or should it just provide generic information during logical planning (e.g. its logical type, is_null(), is_zero()) without access the underlying physical representation?

Physical execution

For physical planning and execution, much like the invocation of UDFs, ExecutionPlan and PhysicalExpr must also be granted access to the LogicalPhysicalType in order to have the capabilities of performing optimised execution for a subset of supported physical types and then fall back to a generic implementation that materialises other types to known physical type. This can be achieved by substituting the use DataType and Schema with, respectively, LogicalPhysicalType and LogicalPhysicalSchema.

Impact on downstream dependencies

Care must be put in place not to introduce breaking changes for downstream crates and dependencies that build on top of DataFusion.

The most impactful changes introduced by this proposal are the LogicalPhysicalType, LogicalPhysicalSchema and LogicalPhysicalField types. These structures would replace most of the mentions of DataType, Schema and Field in the DataFusion codebase. Type sources (TableProvider / TableSource, VarProvider, and ScalarValue) and Logical / ExecutionPlan nodes would be greatly affected by this change. This effect can be mitigated by providing good Into & From implementations for the new types and providing editing existing function arguments and return types as impl Into<LogicalPhysical*>, but it will still break a lot of things.

Case study: datafusion-comet

datafusion-comet is a high-performance accelerator for Apache Spark, built on top of DataFusion. A fork of the project containing changes from this proposal currently compiles without modifications. As more features in this proposal are implemented, namely UDFs Logical Signature, some refactoring might be required (e.g for CometScalarFunction and other functions defined in the codebase). Refer to the draft's TODOs to see what's missing.

Non-goals topics that might be interesting to dive into

While not the primary goals of this proposal, here are some interesting topics that could be explored in the future:

RecordBatches with same logical type but different physical types

Integrating LogicalPhysicalSchema into DataFusion's RecordBatches, streamed from one ExecutionPlan to the other, could be an interesting approach to support the possibility of two record batches having logically equivalent schemas with different underlying physical types. This could be useful in situations where data, stored in multiple pages mapping 1:1 with RecordBatches, is encoded with different strategies based on the density of the rows and the cardinality of the values.

Current work effort

The [Epic] issue tracking this effort can be found at #12622 .

@notfilippo

Thank you @notfilippo -- I think this proposal is well thought out and makes a lot of sense to me.

If we were to implement it I think the benefits for DataFusion would be enormous

From my perspective, the use of Arrow types in logical planning in DataFusion (e.g. type coercion) has always been a little bit of an impedance mismatch. When there were just a few variants (e.g. String/LargeString and Dictionary) it was annoying but manageable.

As Arrow evolves (e.g. to include REEArray, StringViewArray, etc) the mismatch is becoming more painful (e.g. #10220 is an example)

Care must be put in place not to introduce breaking changes for downstream crates and dependencies that build on top of DataFusion.

I think breaking changes to the API is inevitable, but I think we can mange the pain through careful API thought and deprecation. More thoughts to follow

Thoughts on the technical details

This would also mean the introduction of new Field-compatible structure (LogicalPhysicalField)

Since LogicalPlans already use DFSchema, maybe we could avoid introducing a LogialSchema at all and instead update DFSchema to know how to report LogicalType 🤔

Open question: Should ScalarValue split into a LogicalScalarValue and a PhysicalScalarValue? Or should it just provide generic information during logical planning (e.g. its logical type, is_null(), is_zero()) without access the underlying physical representation?

Another possibility wuld be to migate ScalarValue so it was entirely logical (the usecase for a Physical scalar value is not large). We could slowly remove the non-logical versions of ScalarValue (e.g. remove ScalarValue::LargeString and ScalarValue::Dictionary) -- this would give us some idea of where the boundary would be as well as I think improve the usability (we have for example, hit issues where ScalarValue::StringView were treated differently than ScalarValue::Utf8)

I think the biggest challenge of this project will be managing the transition from Arrow DataType to LogicalTypes. You have done a great job articulating the places that would need to be changed. I think we could manage the transition over time by for example deprecating (but leaving UDF APIs in terms of DataType)

ColumnarValue enum could be extended so that functions could choose to provide their own optimised implementation for a subset of physical types and then fall back to a generic implementation that materialises the argument to known physical type.

ANother possibility would be to make a function like ColumnarValue::into_one_of that ensured (via casting if necessary) the input was one of the types supported by the operator. For example

let input: ColumnarValue = &args[0];

// get input as one of the named types, casting if necessary
let input = input.into_one_of(&[DataType::Utf8View, DataType::Utf8])?;

match input.data_type() {
  DataType::Utf8View => { /*specialized impl for StringViewArray */ },
  DataType::Utf8 =>     { /*specialized impl for StringArray */ },
  _ => unreachable!()
}

RecordBatches with same logical type but different physical types

Another thing we could do is relax the requirement that the DataTypes matched exactly between physical plans, and instead just require that the LogicalTypes matches (in other words an ExecutionPlan could report it generates output like Utf8 but could actually produce output like Utf8View 🤔

Since LogicalPlans already use DFSchema, maybe we could avoid introducing a LogialSchema at all and instead update DFSchema to know how to report LogicalType 🤔

Initially, I planned to propose repurposing the DFSchema for this change. Still, I decided against it (at least for the first draft) because of this open question that I've noted above:

Open question: Should we really introduce a new schema type or should we reuse DFSchema? The qualifiers for fields in DFSchema should not be specified by the Tables themselves.

This issue originates from the fact that TableSource and TableProvider (the "native" sources of schemas) would have to return a DFSchema to include the LogicalPhysicalTypes. Adopting DFSchema at that level would mean that both TableSource and TableProvider could edit the qualifiers of the DFSchema, which doesn't seem right (even if the qualifiers would then be replaced or ignored).

@notfilippo

This proposal makes sense to me. Thanks for driving this @notfilippo.

This issue originates from the fact that TableSource and TableProvider (the "native" sources of schemas) would have to return a DFSchema to include the LogicalPhysicalTypes

I was thining that a DFSchema could still wrap an arrow Schema but translate to LogicalType

So like if a TableProvider said it returned a StringView or a Dictionary(Int32, String) the DFSchema would report LogicalType::Utf8 or something

I haven't looked at the code so there may be some reason this wouldn't work

The main challenge I see is that this will be a very large project. The high-level premise of separating logical and physical types makes sense from a first-principle POV, but the cost/benefit analysis at this point is not very clear to me. The latter will probably depend on the implementation strategy and the exact architecture we adopt, so I think we should try out different ideas and approaches before committing to a strategy/architecture and accepting/rejecting/deferring the idea based on a possibly premature choice.

Another possibility wuld be to migate ScalarValue so it was entirely logical (the usecase for a Physical scalar value is not large).

AFAICT ScalarValue is used quite heavily in physical stuff like accumulators and other places. Many "state"-like information we track during the course of execution relies on ScalarValue. Maybe I'm misunderstanding what you meant here?

AFAICT ScalarValue is used quite heavily in physical stuff like accumulators and other places. Many "state"-like information we track during the course of execution relies on ScalarValue. Maybe I'm misunderstanding what you meant here?

I was thinking that there is no fundamental difference between using ScalarValue::String and ScalarValue::LargeString and ScalarValue::Dictionary(..) to accumulate string values as they all eventually wrap String

Thus I was thinking we could simplify the physical implementations by not having different codepaths and this would also give us some first hand experience in how mapping Logical --> Physical types might look like

This generally looks good. I agree that starting with making ScalarValue to represent a logical type would be a good starting point.

One small nit: I don't think I would lump together FixedSizeBinary with Binary and FixedSizeList with List. The fixed lengths often have semantics that should be considered. For example, we use FixedSizeList<f32> as a vector, where the length list is the vector dimension.

Is it possible to have the mapping of the relation between arrow's DataType and LogicalType without TypeRelation?

As long as there is only one LogicalType for each arrow::DataType, we can build up the mapping via From/Into conversion.

I was thining that a DFSchema could still wrap an arrow Schema but translate to LogicalType

We can then easily get the LogicalType from DataType inside Schema.

Is there any type mapping that can't be done without TypeRelation 🤔 ? In other words, is there any arrow::DataType that
can be mapped to more than one LogicalType?

Is there any type mapping that can't be done without TypeRelation 🤔 ? In other words, is there any arrow::DataType that
can be mapped to more than one LogicalType?

I think DataType::Dictionary(..) would map to more than one LogicalType (it would map to whatever the dictionary values type was)

Thus I was thinking we could simplify the physical implementations by not having different codepaths and this would also give us some first hand experience in how mapping Logical --> Physical types might look like

This could be interesting to try -- both in terms of whether we can somehow simplify ScalarValue and also accumulate learnings to help towards this proposal.

It would also be a relatively contained draft, and if the result is satisfactory, can possibly get merged in even if the full proposal is for some reason not yet doable in our current state

It would also be a relatively contained draft, and if the result is satisfactory, can possibly get merged in even if the full proposal is for some reason not yet doable in our current state.

I agree with this approach. I'll dive deeper next week and report back my findings.

Is there any type mapping that can't be done without TypeRelation 🤔 ? In other words, is there any arrow::DataType that
can be mapped to more than one LogicalType?

I think DataType::Dictionary(..) would map to more than one LogicalType (it would map to whatever the dictionary values type was)

Why don't we have LogicalType::Dict(LogicalPhysicalFieldRef), similar to List, Struct, and Map. Although Dict is not really a nested type like those 3, but with LogicalType::Dict(LogicalPhysicalFieldRef) we could probably avoid replacing DFSchema with LogicalPhysicalSchema

or something like

#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub enum LogicalType {
    LogicalPrimitiveType,
    Date,
    Time32(TimeUnit),
    Time64(TimeUnit),
    Timestamp(TimeUnit, Option<Arc<str>>),
    Duration(TimeUnit),
    Interval(IntervalUnit),
    List(LogicalPhysicalFieldRef),
    Struct(LogicalPhysicalFields),
    Dict(LogicalPrimitiveType),
    Map(LogicalPhysicalFieldRef, bool),
    Decimal128(u8, i8),
    Decimal256(u8, i8),
    Union(LogicalUnionFields, UnionMode), // TODO: extension signatures?
    // UserDefinedType
}

pub enum LogicalPrimitiveType {
    Null,
    Int8,
    Int16,
    Int32,
    Int64,
    UInt8,
    UInt16,
    UInt32,
    UInt64,
    Boolean,
    Float16,
    Float32,
    Float64,
    Utf8,
    Binary, 
}

we could probably avoid replacing DFSchema with LogicalPhysicalSchema

That is an interesting idea. I don't fully understand the implications, but if we can figure out how to avoid having to switching from DFSchema I suspect it would make this project less impactful on downstream crates

In physical layer, it is where we need to understand the underlying representation. Then we can treat Json as String and get the StringArray we want.

Yes, I think it would make a lot of sense in the physical layer to simply pass along the actual arrow type (String in this case) as all the function calls / operators will have been resolved to do whatever conversion is needed (e.g. for comparisons)

// function where we care about the internal physical type so we can modify the Array.

ANother way to potentially handle this is that the function call gets resolved to a specific implementation that gets specific Array types (like today). Physical planning / the extension type would be responsible for figuring out which specific implementation to call, etc

We want a one-way mapping from Arrow's DataType to user-defined or extension types, where the logical type (DataFusion's native type) acts as the single source of truth within DataFusion, much like native types in Rust.

I agree with the need for DataFusion's native type to be the single source of truth within DataFusion, to be "the types" of expressions, columns, etc.

We clearly cannot have a one-way mapping / back mapping from Arrow's DataType back to DF Type. Arrow DataType cannot say "it's json" or "it's geometry". We can encode this information in Arrow metadata field and this would be useful for clients reading data from DataFusion. However, this won't be sufficient for intermediate expressions, so the DataFusion types need to be treated as the DF's responsibility, with Arrow types (DataType) handling as carrier type / physical type only.

confusing was ExtensionType::logical_type in your example returns a LogicalType
pub trait ExtensionType {
    fn logical_type(&self) -> LogicalType;
}

I see this could be seen as a solution to some problems like function calls (vide simple functions #12635).
If we can generate function calls for various string array types for a function accepting string, then we should be able to do the same for a function accepting JSON. I agree, however, that we need to express this differently, i.e. [logical] type should not redirect to another [logical] type for its implementation, otherwise we will force ourselves into a bad design with first-class and second-class types.

BTW what if this LogicalType / Type thing was not an enum at all, just a trait?

We clearly cannot have a one-way mapping / back mapping from Arrow's DataType back to DF Type. Arrow DataType cannot say "it's json" or "it's geometry". We can encode this information in Arrow metadata field and this would be useful for clients reading data from DataFusion. However, this won't be sufficient for intermediate expressions, so the DataFusion types need to be treated as the DF's responsibility, with Arrow types (DataType) handling as carrier type / physical type only.

I still don't get the point why not, is there any example that we require more than one Logical Type for DataType?

Given Arrow's extension type like Json, we can consider it as either LogicalType::UserDefined(Json) or LogicalType::String if you want (it is customizable by design), but it does not make sense to have both cases.

If the one-way mapping doesn't work, we can introduce able_to_decode for the DataType that returns Vec of LogicalType. If any matched, we can decode the value to the target DataType.

BTW what if this LogicalType / Type thing was not an enum at all, just a trait?

Trait for user defined type is much more flexible. For native type we can have either trait or enum.

I still don't get the point why not, is there any example that we require more than one Logical Type for DataType?

if we allow types to be extended (#12644), many different types may use same arrow type as their carrier type (eg JSON -> String, Geo -> String).

Given Arrow's extension type like Json, we can consider it as either LogicalType::UserDefined(Json) or LogicalType::String if you want (it is customizable by design), but it does not make sense to have both cases.

logical JSON and logical String may both use Arrow String, but they have different behavior, e.g. when casting from/to some other type.

If the one-way mapping doesn't work, we can introduce able_to_decode for the DataType that returns Vec of LogicalType.

i may be naive here, but i do very much hope we need only unidirectional mapping, from logical types to arrow types.
We need to retain information what the logical/DF type was (eg when returning data to the client, so they know too), but we should might not need this information at runtime.

BTW what if this LogicalType / Type thing was not an enum at all, just a trait?

Trait for user defined type is much more flexible. For native type we can have either trait or enum.

Yes, both impl strategies seem syntactically equivalent and starting from enum feels like a lazy choice.
I am experimenting with trait-as-first-class-citizen approach and see how this works.
(This would be similar in spirit to unifying buildin and extension UDAFs under single trait #8708)

if we allow types to be extended (#12644), many different types may use same arrow type as their carrier type (eg JSON -> String, Geo -> String).

I think this is what we have not consensus on, for arrow extension type, I think they should be considered as an individual type. Json, Geo, String are 3 different arrow data type, thus 3 different logical types. Even they all have StringArray internally but the metadata or the rule to describe the StringArray should be considered different, otherwise why we not just use StringArray at the first place?

For logical type, they can have the same DataType, logical JSON and logical String may both use Arrow String and they can both cast to StringArray, and we can still tell the difference between these 2 logical type and process them with different logic.

I think my assumption should be changed to At most one DatafusionNativeType for each DataType instead. We can also think like this, we can find the minimum type set (DatafusionNativeTypes) and all the Arrow DataType including extension type and the LogicalType including user defined type can be mapped to at most one type in the DatafusionNativeTypes

If the one-way mapping doesn't work, we can introduce able_to_decode for the DataType that returns Vec of LogicalType. If any matched, we can decode the value to the target DataType.

Maybe what we need is just checking whether the LogicalType we have is able to decode to the certain Arrow DataType 🤔 In this case we just need to know the set of types

trait LogicalType {
    fn can_decode_to(&self, data_type: DataType) -> bool;
}

struct String {}
impl LogicalType for String {
    fn can_decode_to(&self, data_type: DataType) -> bool {
        matches!(data_type, DataType::Utf8 | DataType::Utf8View | DataType::LargeUtf8)
    }
}

struct Json {}
impl LogicalType for Json {
    fn can_decode_to(&self, data_type: DataType) -> bool {
        matches!(data_type, DataType::Utf8 | DataType::Utf8View | DataType::LargeUtf8)
    }
}

struct StringViewOnly {}
impl LogicalType for StringViewOnly {
    fn can_decode_to(&self, data_type: DataType) -> bool {
        matches!(data_type, DataType::Utf8View)
    }
}
struct Geo {}
impl LogicalType for Geo {
    fn can_decode_to(&self, data_type: DataType) -> bool {
        if let DataType::FixedSizeList(inner_type, n) = data_type {
            return matches!(inner_type.data_type(), DataType::Float32 | DataType::Float64) && n > 1
        }

        return false;
    }
}

struct String {}
impl LogicalType for String {
    fn can_decode_to(&self, data_type: DataType) -> bool {
        matches!(data_type, DataType::Utf8 | DataType::Utf8View | DataType::LargeUtf8)
    }
}

I don't really like that this function would require recursion to handle Dictionary(Dictionary(...(Utf8))) but it's definitely an interesting approach.

@jayzhan211

@jayzhan211 and @findepi I really like the approach of LogicalType as a trait and I was able to validate it by using an old working branch with the original proposal code. I ended up using this:

pub trait LogicalType {
    fn native(&self) -> &NativeType;
    fn name(&self) -> Option<&str>; // as in 'ARROW:extension:name'.
}

pub struct String {}

impl LogicalType for String {
    fn native(&self) -> &NativeType {
        &NativeType::Utf8
    }

    fn name(&self) -> Option<&str> {
        None
    }
}

pub struct JSON {}

pub const JSON_EXTENSION_NAME: &'static str = "arrow.json";

impl LogicalType for JSON {
    fn native(&self) -> &NativeType {
        &NativeType::Utf8
    }

    fn name(&self) -> Option<&str> {
        Some(JSON_EXTENSION_NAME)
    }
}

And by modifying the temporary TypeRelation trait in order to keep track of both physical and logical types in the whole datafusion system:

pub trait TypeRelation {
    fn logical(&self) -> &dyn LogicalType;
    fn physical(&self) -> &DataType;
}

pub struct NativeTypeRelation {
    logical: Arc<dyn LogicalType>,
    physical: DataType,
}

impl TypeRelation for NativeTypeRelation {
    fn logical(&self) -> &dyn LogicalType {
        &*self.logical
    }

    fn physical(&self) -> &DataType {
        &self.physical
    }
}

impl From<DataType> for NativeTypeRelation {
    fn from(value: DataType) -> Self {
        let logical = match value {
            DataType::Utf8 | DataType::Utf8View | DataType::LargeUtf8 => {
                Arc::new(String {})
            }
            _ => todo!(),
        };
        Self {
            logical,
            physical: value,
        }
    }
}

I was able to run pass some string tests and write "JSON specific" functions.

TypeRelation

Do we still need TypeRelation? Where is it used?

TypeRelation
Do we still need TypeRelation? Where is it used?

For the final state of this proposal: no. As a transitory type to validate the concept you described above: yes

@notfilippo

pub trait LogicalType {
    fn native(&self) -> &NativeType;
    fn name(&self) -> Option<&str>; // as in 'ARROW:extension:name'.
}

@notfilippo This might work. I started similar prototype for #12644.

To reliably encode types in Arrow schema, we would need a structured approach, perhaps something like this

pub trait Type: Sync + Send {
    fn name(&self) -> &Arc<TypeName>;
...
}

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TypeName {
    basename: String,
    name: String,
    parameters: Vec<TypeParameter>,
}

pub type TypeNameRef = Arc<TypeName>;

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum TypeParameter {
    Type(TypeNameRef),
    Number(i128),
}

impl TypeName {
    pub fn basename(&self) -> &str {
        &self.basename
    }
    pub fn name(&self) -> &str {
        &self.name
    }
    pub fn parameters(&self) -> &[TypeParameter] {
        &self.parameters
    }

    pub fn new(basename: impl Into<String>, parameters: Vec<TypeParameter>) -> Self {
        // TODO assert parses, no braces, etc.
        let basename = basename.into();
        let name = FormatName {
            basename: &basename,
            parameters: &parameters,
        }
        .to_string();
        Self {
            basename,
            name,
            parameters,
        }
    }
    pub fn from_basename(basename: impl Into<String>) -> Self {
        Self::new(basename, vec![])
    }
}

To allow type-checking ("is int" or "is string") for optimizer rules such as UnwrapCastExprRewriter we could use Any

pub trait Type: Sync + Send {
    fn as_any(&self) -> &dyn Any;
...
}

Type should also describe how to compare (native) values of that type, their equality, etc.
(for example, per SQL spec, timestamp with time zone has interesting equality defined on part of the value, so bit-by-bit equality cannot be assumed).

@notfilippo

@notfilippo however, i realized that changing the logical type system would be easier if we had clearly defined architectural layers. I filed #12723 for this.

Discussion continues under #12622. Let me close this one. I don't think there is a benefit of keeping two issues open for single topic, even if the topic is as fundamental as this one.

Navigation Menu

Repository navigation

[Proposal] Decouple logical from physical types #11513

Abstract

Goals

Proposal

Defining a logical type

User defined types

User defined physical types

User defined logical types

Boundaries of logical and physical types

In plans and expressions

In functions

In substrait

Keeping track of the physical type

A new type of Schema

Type sources

Tables

VarProvider

ScalarValue

Physical execution

Impact on downstream dependencies

Case study: datafusion-comet

Non-goals topics that might be interesting to dive into

RecordBatches with same logical type but different physical types

Current work effort

38 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Proposal] Decouple logical from physical types #11513

Description

Abstract

Goals

Proposal

Defining a logical type

User defined types

User defined physical types

User defined logical types

Boundaries of logical and physical types

In plans and expressions

In functions

In substrait

Keeping track of the physical type

A new type of Schema

Type sources

Tables

VarProvider

ScalarValue

Physical execution

Impact on downstream dependencies

Case study: datafusion-comet

Non-goals topics that might be interesting to dive into

RecordBatches with same logical type but different physical types

Current work effort

Activity

alamb commented on Jul 19, 2024

alamb commented on Jul 19, 2024

notfilippo commented on Jul 19, 2024

andygrove commented on Jul 19, 2024

alamb commented on Jul 19, 2024

ozankabak commented on Jul 19, 2024

ozankabak commented on Jul 19, 2024

alamb commented on Jul 19, 2024

wjones127 commented on Jul 19, 2024

jayzhan211 commented on Jul 20, 2024

alamb commented on Jul 20, 2024

ozankabak commented on Jul 20, 2024

notfilippo commented on Jul 20, 2024

jayzhan211 commented on Jul 20, 2024

alamb commented on Jul 21, 2024

38 remaining items

alamb commented on Sep 29, 2024

alamb commented on Sep 29, 2024

findepi commented on Sep 30, 2024

jayzhan211 commented on Sep 30, 2024

findepi commented on Sep 30, 2024

jayzhan211 commented on Sep 30, 2024

jayzhan211 commented on Sep 30, 2024

notfilippo commented on Sep 30, 2024

notfilippo commented on Sep 30, 2024

jayzhan211 commented on Oct 3, 2024

notfilippo commented on Oct 3, 2024

findepi commented on Oct 9, 2024

findepi commented on Oct 9, 2024

findepi commented on Feb 10, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions