Blog post about using external indexes with Parquet

@zhuqi-lucas

Is your feature request related to a problem or challenge?

I would like to write a blog that explains how to use DataFusion and external indexes when reading parquet files

We have some existing example which I think shows how to make them work, lacks sufficient context I think for a wider audience

https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/advanced_parquet_index.rs
https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/parquet_index.rs

Describe the solution you'd like

I recently did a talk ( video and presentation) where I explained a lot of this, which I think can be turned into a blog pretty reasonably

Describe alternatives you've considered

No response

Additional context

fyi @zhuqi-lucas and @JigaoLuo may also be interested

@alamb

Thank you @alamb, great idea, i can help review and contribute.

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Good idea , may be we can add simple clickbench example, to rewrite with some external indexes, and the performance becomes better?

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Good idea , may be we can add simple clickbench example, to rewrite with some external indexes, and the performance becomes better?

Yes, exactly. I’ll also go through the video and the draft blog—hopefully that’ll help me come up with something more concrete.

Thanks! I just skimmed through it quickly. I think it’d be interesting to run an experiment with plots comparing performance with and without the specific indexing to evaluate its impact. I’ll try to carve out some time for it—definitely excited to see what we could find!

Here is a crazy specific idea:

I think something that would be broadly interesting (aka get a log of clicks / likely to make Hacker news) would be a some example of fast single row lookups with Parquet -- this is a common usecase that is cited by new file formats

The narrative could go something like

Parquet is often used for analytical, scan heavy queries and the built in indexing structures and writer properties are optimized for this case
However, you can combine external indexes and the appropriate writer properties (e.g 100 row pages so the overhead of decoding a page is low) to build a system that works well for single row lookups
Look we prototyped an example using DataFusion that can do single row lookups in under 1ms with the data is locally in cache

If I’m not mistaken, that aligns pretty well with my understanding of Lance v2.
Also, I’m not sure if it’s a good idea, but could we try reproducing this experiment setup with a Parquet with a special index — or adapt it: https://blog.lancedb.com/benchmarking-random-access-in-lance/

The blog is ready for review

PR:

[BLOG] Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet datafusion-site#99

Rendered Preview: https://datafusion.staged.apache.org/blog/2025/08/15/external-parquet-indexes/

And the post is live:

https://datafusion.apache.org/blog/2025/08/15/external-parquet-indexes/

@alamb

Hi @alamb

Thanks so much for inviting me and for sharing the post on LinkedIn! I really appreciate it. https://www.linkedin.com/posts/apache-datafusion_using-external-indexes-metadata-stores-activity-7362130830053289984-nBag?utm_source=share&utm_medium=member_desktop&rcm=ACoAACyRbsYBcsi6ISDnXZC-xALLzcpTAPF-LHQ

I just wanted to mention—there’s a small typo in my name in the post. If it’s possible to update it, that would be great. Also, here’s my LinkedIn page for reference: https://www.linkedin.com/in/jigao-luo/ Thanks again

Yes, of course . I am very sorry about that. Do you mean this one? I tried really hard to copy/paste from github and I don't see the typo. I must be missing something

I am very 🤦 sorry about that.

No. I mean the LinkedIn thing above and also here:

@JigaoLuo

Ah, thank you @JigaoLuo -- I will fix that asap

Fixed

Navigation Menu

Repository navigation

Blog post about using external indexes with Parquet #17010

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Blog post about using external indexes with Parquet #17010

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

zhuqi-lucas commented on Aug 1, 2025

JigaoLuo commented on Aug 1, 2025

zhuqi-lucas commented on Aug 1, 2025

JigaoLuo commented on Aug 1, 2025

alamb commented on Aug 1, 2025

JigaoLuo commented on Aug 1, 2025

alamb commented on Aug 8, 2025

alamb commented on Aug 15, 2025

JigaoLuo commented on Aug 15, 2025

alamb commented on Aug 15, 2025

JigaoLuo commented on Aug 15, 2025

alamb commented on Aug 15, 2025

alamb commented on Aug 15, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions