Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives "type16 not valid error" when schema includes date32[day] type #949

Activity

1. preetijoshi-womply

   added

   bugSomething isn't workingSomething isn't working

   on Aug 26, 2021
2. preetijoshi-womply

   changed the title [-]Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives schema error[/-] [+]Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives "type16 not valid error" when schema includes date32[day] type[/+] on Aug 26, 2021
3. drauschenbach commented on Nov 3, 2024

   

   drauschenbach

   on Nov 3, 2024

   Contributor

   More actions

   This issue appears to have been resolved.

   import datafusion
   import pyarrow
   import datetime
   
   ctx = datafusion.SessionContext()
   
   batch = pyarrow.RecordBatch.from_arrays(
       [pyarrow.array([1, 2, 3]), pyarrow.array([datetime.date(1970, 1, 1),datetime.date(1970, 1, 2),datetime.date(1970, 1, 3)])],
       names=["a", "b"],
   )
   df = ctx.create_dataframe([[batch]])
   
   
   >>> print(df)
   DataFrame()
   +---+------------+
   | a | b          |
   +---+------------+
   | 1 | 1970-01-01 |
   | 2 | 1970-01-02 |
   | 3 | 1970-01-03 |
   +---+------------+
   
   
   >>> print(df.schema())
   a: int64
   b: date32[day]
4. andygrove commented on Nov 3, 2024

   

   andygrove

   on Nov 3, 2024

   Member

   More actions

   Closing this. Thanks @drauschenbach.
5. andygrove

   closed this as completedon Nov 3, 2024
6. HairstonE

   added a commit that references this issue on Oct 7, 2025

   Add make_list and tests for make_list, make_array (apache#949)

   92b093c