Repository navigation
Browser-accessible official DataFusion playground / DataFusion fiddle #13818
Description
Activity
💯 Thank you @findepi
Here is some inspiration:
https://duckdb.org/docs/api/wasm/overview.html
It would be amazing to have something similar in the datafuion doc site. Something like this:
Reacted by Piotr FindeisenAs @findepi points out there are at least two implementations of something similar already on #9834
These from @waynexia :
- Demo: https://datafusion-contrib.github.io/datafusion-wasm-playground/
- Source: https://github.com/datafusion-contrib/datafusion-wasm-playground
- https://github.com/datafusion-contrib/datafusion-wasm-bindings
These from @XiangpengHao
This should greatly reduce the bar of trying out DataFusion 👍
Actually I have tried the "black terminal" style before, as the first version:
It turns out that this style requires a lots of logic in the frontend side to simulate a terminal, which is why I change it to the
datafusion-wasm-playgroundlooks like now. But this is not the most important thing, just to mention it.Reacted by Piotr Findeisen and Andrew LambIt would be useful to be able to quickly test queries in current DataFusion
That's exactly why I started parquet-viewer -- to view paruqet files and experiment with queries without using the cli tool.
If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)
parquet viewer is super cool, especially when a user starts with some data in the form of parquet.
I was thinking about something more SQL centric. Most discussions in issues and PRs revolve around SELECT with perhaps CREATE TABLE, etc. It would be nice to be able to run those on current main or last released version, without having to create Parquet file first.@XiangpengHao Do you think this can be evolved into simple query editor with SELECT and CREATE TABLE capabilities, where Parquet upload is an option to create a table?
Maybe we are looking for something like this? https://sql-workbench.com
currently parquet viewer is completely in Rust, and we create js binding when we need to interact with browser, so it can be quite challenging to incorporate to many fancy frontend lib, like monaco editor etc...
If we want some cool editor experience, it's better to create a binding of DataFusion and develop from the frontend side, like what @waynexia has done for datafusion-wasm-bindings.
But my frontend knowledge can barely understand what llm generates...If people like it, I can donate it to a more official-looking repo and make it a starting point of datafusion online (or something similar)
For the donate part, how about making it the new datafusion-playground. Parquet viewer has implemented a great function set with a more beautiful appearance. Actually I don't want to maintain the old frontend project after seeing the viewer 🙈
Reacted by Andrew Lamb- changed the title
[-]Browser-accessible official DataFusion playground[/-][+]Browser-accessible official DataFusion playground / DataFusion fiddle[/+]on Feb 4, 2025 I just game across one from @gabotechs : https://datafusion-fiddle.vercel.app/
Repo: https://github.com/gabotechs/datafusion-fiddle
Reacted by GabrielReacted by Piotr FindeisenReacted by Ruihang XiaMade it a while ago and still use it for quickly trying out stuff, there are lots of low hanging fruits for improving it (local storage based query history, syntax/error highlighting, a proper open source table component, etc...). If there's something I can do to help please let me know.
This looks very cool, @gabotechs! And even better than the CLI itself (#13286) :)
I took the privilege to file the first enhancement idea - gabotechs/datafusion-fiddle#1.Reacted by GabrielI'm imaging a highly encapsulated JS component like
<DataFusion config={config}, interactive=true />
and the user who wants an interactive terminal can render a DataFusion webpage with one line, who wants to use DataFusion WASM to do computation can invoke it as a JS function
Reacted by Andrew LambHi @XiangpengHao and @waynexia is there a way to make the a website like in parquet-viewer without reading the whole file from s3 like you implemented into the memory or are you saving it into filesystem?
because as far as I know its not possible to read and write to filesystem in wasm as of yet?Hi @XiangpengHao and @waynexia is there a way to make the a website like in parquet-viewer without reading the whole file from s3 like you implemented into the memory or are you saving it into filesystem? because as far as I know its not possible to read and write to filesystem in wasm as of yet?
Parquet viewer does not read the whole file; it only reads the metadata and data relevant to your query.
You almost instantly get the query results even if the parquet file is a few GBs large.
Edit: caching is done by the browser
Reacted by Andrew Lamb, Kyle Barron and pranavJibhakateThanks a lot I will look into the implementation of parquet-viewer about how the caching is done.
Thanks a lot I will look into the implementation of parquet-viewer about how the caching is done.
This is the in-memory cache implementation: https://github.com/XiangpengHao/parquet-viewer/blob/main/src/object_store_cache.rs
Disk cache is done automatically by the browser, so it may or may not work depends on many things (request size, range, browser settings, freshness, etc)
Hi everyone,
I've opened a discussion to gather ideas on the design and features we’d like to see in the official DataFusion playground.
Would love to hear your thoughts—feel free to jump in and share your ideas. Thanks in advance! 😊Reacted by Andrew Lambhttps://datafusion-fiddle.vercel.app/ exists and is awesome. I am closing the issue.
Note the site currently features v45.0.0. Would be great to have new fusion versions auto-deployed there.
However, I think it's not something DF project has currently bandwidth to take responsibility for.Reacted by Andrew Lambhttps://datafusion-fiddle.vercel.app/ exists and is awesome. I am closing the issue.
Note the site currently features v45.0.0. Would be great to have new fusion versions auto-deployed there. However, I think it's not something DF project has currently bandwidth to take responsibility for.
Looks like @gabotechs is the owner of https://github.com/gabotechs/datafusion-fiddle
If you want to move the app into the
datafusion-contriborganization https://github.com/datafusion-contrib @gabotechs I would be happy to give you the permissions to do soMoving it into that organization might make it easier for other people to find and contribute to
Reacted by Piotr Findeisen and GabrielMakes sense, I'd be happy to move it there. The app is hosted in vercel now connected to my Github account, not sure if there's something else to be done about that apart from just putting it into datafusion-contrib.
Makes sense, I'd be happy to move it there. The app is hosted in vercel now connected to my Github account, not sure if there's something else to be done about that apart from just putting it into datafusion-contrib.
Let's start just by moving the code into datafusion-contrib (I sent you an invite) and if it gets more attention and people want to help move it to some other hosting platform (like github pages or something) that might be a nice follow on contribution
I'll invite you to the org
Reacted by GabrielMoved, it's already on https://github.com/datafusion-contrib/datafusion-fiddle. I made a small revamp before moving it, so it should look a bit better now.
Note: deployments won't work unless a datafusion-contrib owner accepts linking datafusion-contrib/datafusion-fiddle to my vercel account (a request should have already been sent) or a new vercel account is created and linked to the project. I'd be happy to still host the project in my vercel account, but if you think there's a better way that also works.
Reacted by Andrew Lamb and Kyle BarronMoved, it's already on https://github.com/datafusion-contrib/datafusion-fiddle. I made a small revamp before moving it, so it should look a bit better now.
Nice! it is also at DataFusion 48
Note: deployments won't work unless a datafusion-contrib owner accepts linking datafusion-contrib/datafusion-fiddle to my vercel account (a request should have already been sent) or a new vercel account is created and linked to the project. I'd be happy to still host the project in my vercel account, but if you think there's a better way that also works.
Done!
Thank you
Reacted by Gabriel
Is your feature request related to a problem or challenge?
It would be useful to be able to quickly test queries in current DataFusion (latest release or nightly snapshot), without having to check out main branch and running the build.
It would lower the barrier of entry to verifying repro steps in issues, etc.
Describe the solution you'd like
a simple, official website hosted by the DataFusion project allowing to run queries using WASM build of DataFusion
Describe alternatives you've considered
cargo run
Not doing this and requiring contributors to instead
This is time consuming, and discourages issue reporters from checking their repro steps.
Or project maintainers from verifying things in current main.
docker image
Having datafusion docker image and using that.
This would be pretty convenient for maintainers, but not less accessible for the contributors or users. Browser-based would be more accessible
Additional context