Repository navigation
Add example of catalog API usage (#5291) - #5326
Conversation
alamb
left a comment
There was a problem hiding this comment.
I ran cargo run --example catalog
querying table delta_binary_packed.parquet from parquet schema
+---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+---------------+---------------+---------------+----------------+----------------+----------------+----------------+-----------------+-----------------+-----------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+---------------------+---------------------+---------------------+----------------------+----------------------+----------------------+----------------------+-------------+
| bitwidth0 | bitwidth1 | bitwidth2 | bitwidth3 | bitwidth4 | bitwidth5 | bitwidth6 | bitwidth7 | bitwidth8 | bitwidth9 | bitwidth10 | bitwidth11 | bitwidth12 | bitwidth13 | bitwidth14 | bitwidth15 | bitwidth16 | bitwidth17 | bitwidth18 | bitwidth19 | bitwidth20 | bitwidth21 | bitwidth22 | bitwidth23 | bitwidth24 | bitwidth25 | bitwidth26 | bitwidth27 | bitwidth28 | bitwidth29 | bitwidth30 | bitwidth31 | bitwidth32 | bitwidth33 | bitwidth34 | bitwidth35 | bitwidth36 | bitwidth37 | bitwidth38 | bitwidth39 | bitwidth40 | bitwidth41 | bitwidth42 | bitwidth43 | bitwidth44 | bitwidth45 | bitwidth46 | bitwidth47 | bitwidth48 | bitwidth49 | bitwidth50 | bitwidth51 | bitwidth52 | bitwidth53 | bitwidth54 | bitwidth55 | bitwidth56 | bitwidth57 | bitwidth58 | bitwidth59 | bitwidth60 | bitwidth61 | bitwidth62 | bitwidth63 | bitwidth64 | int_value |
+---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+---------------+---------------+---------------+----------------+----------------+----------------+----------------+-----------------+-----------------+-----------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+---------------------+---------------------+---------------------+----------------------+----------------------+----------------------+----------------------+-------------+
| 6374628540732951412 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | -2070986743 |
| 6374628540732951412 | -1 | -2 | -4 | -8 | -16 | -32 | -64 | -128 | -256 | -512 | -1024 | -2048 | -4096 | -8192 | -16384 | -32768 | -65536 | -131072 | -262144 | -524288 | -1048576 | -2097152 | -4194304 | -8388608 | -16777216 | -33554432 | -67108864 | -134217728 | -268435456 | -536870912 | -1073741824 | -2147483648 | -4294967296 | -8589934592 | -17179869184 | -34359738368 | -68719476736 | -137438953472 | -274877906944 | -549755813888 | -1099511627776 | -2199023255552 | -4398046511104 | -8796093022208 | -17592186044416 | -35184372088832 | -70368744177664 | -140737488355328 | -281474976710656 | -562949953421312 | -1125899906842624 | -2251799813685248 | -4503599627370496 | -9007199254740992 | -18014398509481984 | -36028797018963968 | -72057594037927936 | -144115188075855872 | -288230376151711744 | -576460752303423488 | -1152921504606846976 | -2305843009213693952 | -4611686018427387904 | -9223372036854775808 | -22783326 |
| 6374628540732951412 | -1 | -1 | -1 | -5 | -15 | -3 | -3 | -65 | -135 | -151 | -884 | -498 | -541 | -2357 | -13235 | -5032 | -53240 | -29011 | -250328 | -472084 | -14542 | -1696277 | -1840215 | -3928478 | -7271486 | -21651755 | -32837885 | -111433329 | -241601040 | -50065479 | -163270799 | -1986952430 | -4199967586 | -1344496374 | -16288562574 | -28644417946 | -65240709120 | -70538406032 | -191624370730 | -296605683166 | -1011770835153 | -1608485630598 | -4014593275381 | -6105818392192 | -16882699520562 | -24806559006505 | -2633546645122 | -117376993649456 | -108462505318873 | -17587725526794 | -404921059206593 | -156565041626495 | -314837785970473 | -5814847697611025 | -12226190471490179 | -25585079015954934 | -15550441703940602 | -56208863042127981 | -90038694817934139 | -119581030183793765 | -506045522191644725 | -608563416973139311 | -268654784870221847 | -725202170854031360 | -1782018724 |
| 6374628540732951412 | -1 | -3 | -3 | -3 | -14 | 19 | 9 | -70 | -291 | -145 | -275 | -779 | -4047 | -8244 | -9036 | 19242 | -48154 | 57337 | -76872 | -140006 | -606577 | -1112970 | -1741237 | -9834470 | -18759951 | -6702647 | -48746571 | -75629270 | -261813622 | -372076116 | -209790348 | -1454655246 | 14148394 | 3594034401 | -22765140849 | -37447824121 | 973131962 | -184009634540 | -414171928390 | -33166024765 | -1197833965973 | -2945889030754 | -4176475312470 | -3485626798002 | -33267360685874 | -30046841485160 | -44798370319938 | -169294344644831 | 43524599724868 | 469339429031087 | -249413706008860 | 1783640864794895 | 2532719170052087 | -6021542079991477 | -22672611942859889 | 1385417775479117 | 25457797741384317 | 15212409470706824 | -293860965757446445 | -421796672552175066 | -739907890298874818 | -359496579899035406 | 1767081710130543267 | -9103419799827896320 | -795597708 |
| 6374628540732951412 | -1 | -2 | -7 | -7 | -23 | -4 | 51 | -107 | -322 | -69 | -404 | -497 | -1478 | -1446 | 3148 | -5651 | -72427 | 2324 | -180065 | -73164 | -1070042 | -2913370 | -2556489 | -13152247 | -18307794 | -21400650 | -21970468 | -49244077 | -154493129 | -865963106 | -675026787 | 269282718 | -1936781439 | 11152842629 | -8330016935 | -20375075134 | 4843904084 | -154956898562 | -348410255803 | -57366930449 | -1692439856605 | -1475642885659 | -8535248964530 | -6025906219756 | -16857628299601 | -62685577005613 | -27221312319071 | -198785994498891 | 171238115176440 | 660593922279105 | -943597583604423 | 2827707839263247 | 4070733887796967 | 1851107790825924 | -10035362931120011 | -34178564238813898 | -15400877254369937 | 14534187603609527 | -436091502934230264 | 103753242946746509 | -433954430332814130 | -2277734064379320830 | -1952725388208511715 | -598707373574004736 | -50404127 |
+---------------------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+-------------+-------------+-------------+-------------+--------------+--------------+--------------+---------------+---------------+---------------+----------------+----------------+----------------+----------------+-----------------+-----------------+-----------------+------------------+------------------+------------------+-------------------+-------------------+-------------------+-------------------+--------------------+--------------------+--------------------+---------------------+---------------------+---------------------+----------------------+----------------------+----------------------+----------------------+-------------+
querying table alltypes_plain.parquet from parquet schema
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| id | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col | date_string_col | string_col | timestamp_col |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
| 4 | true | 0 | 0 | 0 | 0 | 0 | 0 | 30332f30312f3039 | 30 | 2009-03-01T00:00:00 |
| 5 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30332f30312f3039 | 31 | 2009-03-01T00:01:00 |
| 6 | true | 0 | 0 | 0 | 0 | 0 | 0 | 30342f30312f3039 | 30 | 2009-04-01T00:00:00 |
| 7 | false | 1 | 1 | 1 | 10 | 1.1 | 10.1 | 30342f30312f3039 | 31 | 2009-04-01T00:01:00 |
| 2 | true | 0 | 0 | 0 | 0 | 0 | 0 | 30322f30312f3039 | 30 | 2009-02-01T00:00:00 |
+----+----------+-------------+--------------+---------+------------+-----------+------------+------------------+------------+---------------------+
querying table null_list.parquet from parquet schema
+-----------+
| emptylist |
+-----------+
| [] |
+-----------+
querying table binary.parquet from parquet schema
+-----+
| foo |
+-----+
| 00 |
| 01 |
| 02 |
| 03 |
| 04 |
+-----+
querying table fixed_length_decimal.parquet from parquet schema
+-------+
| value |
+-------+
| 1.00 |
| 2.00 |
| 3.00 |
| 4.00 |
| 5.00 |
+-------+
dropping table delta_binary_packed.parquet
Compilation finished at Sat Feb 18 11:26:43
👍 looks very nice
|
The CI failure seems unrelated to this PR so merging this one in. THanks @jaylmiller https://github.com/apache/arrow-datafusion/actions/runs/4212165207/jobs/7311048020 |
|
Benchmark runs are scheduled for baseline = a7a824a and contender = 5ffa8d7. 5ffa8d7 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
#5291 (does not close)
Rationale for this change
A simple example can help users figure out how to use DataFusion's catalog API.
What changes are included in this PR?
Just the example.
Are these changes tested?
n/a
Are there any user-facing changes?
n/a