Split EmptyExec into OneRowExec #8355

Activity

1. crepererum

   added

   enhancementNew feature or requestNew feature or request

   on Nov 29, 2023
2. alamb

   added

   good first issueGood for newcomersGood for newcomers

   on Nov 29, 2023
3. alamb commented on Nov 29, 2023

   

   alamb

   on Nov 29, 2023

   Contributor

   More actions

   I think this is a reasonable first (maybe more like second) issue -- it is fairly mechanical but will require figuring out some of how ExecutionPlan works and updating a bunch of tests
4. alamb

   changed the title [-]Split `EmptyExec`[/-] [+]Split `EmptyExec` into `OneRowExec`[/+] on Nov 29, 2023
5. tustvold commented on Nov 29, 2023

   

   tustvold

   on Nov 29, 2023

   Contributor

   More actions

   FWIW unless the single-row aspect of OneRowExec is important, it could possibly just use MemoryExec - https://docs.rs/datafusion/latest/datafusion/physical_plan/memory/struct.MemoryExec.html

   👍React with 👍1Reacted by Marco Neumann
6. alamb commented on Nov 29, 2023

   

   alamb

   on Nov 29, 2023

   Last edited by alamb

   Contributor

   More actions

   single_row is used for queries like select now()

   ❯ select now();
   +-----------------------------+
   | now()                       |
   +-----------------------------+
   | 2023-11-29T15:46:02.712512Z |
   +-----------------------------+
   1 row in set. Query took 0.030 seconds.

   I think we can now make a batch now with no columns but 1 row via https://docs.rs/arrow/latest/arrow/array/struct.RecordBatch.html#method.try_new_with_options which was not always the case . Using MemoryExec might work just fine.

   👍React with 👍1Reacted by Marco Neumann
7. razeghi71 commented on Dec 6, 2023

   

   razeghi71

   on Dec 6, 2023

   Contributor

   More actions

   I have tried to move the produce_one_row to MemoryExec in #8412, but it wasn't straight forward as there are no try_into_physical_plan and try_from_physical_plan implemented for MemoryExec.

   Anyway I have tried to implement the physical plan, but since MemoryExec contains dyn Array for storing it's partitions it's a little bit tricky to serialize it to proto, the best way coming to my mind is to json serialize it as a string field in proto.

   I can do that, but before I wanted to get feedback here and make sure if that's a good path to go? @alamb @tustvold @crepererum
8. alamb commented on Dec 6, 2023

   

   alamb

   on Dec 6, 2023

   Contributor

   More actions

   I have tried to move the produce_one_row to MemoryExec in #8412, but it wasn't straight forward as there are no try_into_physical_plan and try_from_physical_plan implemented for MemoryExec.

   Anyway I have tried to implement the physical plan, but since MemoryExec contains dyn Array for storing it's partitions it's a little bit tricky to serialize it to proto, the best way coming to my mind is to json serialize it as a string field in proto.

   I can do that, but before I wanted to get feedback here and make sure if that's a good path to go? @alamb @tustvold @crepererum

   The serialization is a good point

   What about perhaps making a new OneRowExec instead of reusing MemoryExec ? I think the code would be fairly straightforward.

   👍React with 👍1Reacted by Mohammad Razeghi
9. alamb

   closed this as completedin #8446on Dec 9, 2023