Repository navigation
Any reason why logical plan is optimized twice? #705
Description
Hello, I am just curious why the logical plan is being optimized twice.
First, it was optimized here:
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/context.rs#L205
Then it was optimized again when collecting result
https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/dataframe_impl.rs#L147
Activity
houqp commentedon Jul 11, 2021
probably an oversight? the optimize in sql method does look like redundant.
Dandandan commentedon Jul 11, 2021
I think it's not an oversight:
collectis not necessarily called when executing queries, for example when writing the results to disk. Ascollectloads the result into memory in one thread/partition, this is not always what you want to do.We might do something to avoid optimizing twice when calling both functions, but maybe it's not a big deal for now?
houqp commentedon Jul 11, 2021
That's a good point. Perhaps the better structure would be to push optimize into query evaluation/materialization methods like
saveandcollect. This way, we can make sure the engine always execute with an optimized plan regardless whether it's built from sql or plan builder.alamb commentedon Jul 16, 2021
I like that plan
alamb commentedon Aug 4, 2021
I wonder if this ticket is tracking anything useful or if the question has been answered and we can close the ticket?
houqp commentedon Aug 5, 2021
i think we already got the answer to the question.
fix: dictionary decimal vector optimization (apache#705)