Skip to content

Release Datafusion 6.0.0 #890

Closed
Closed
Release Datafusion 6.0.0#890
Assignees
houqp
Labels
enhancementNew feature or request
Milestone
6.0.0

Description

@houqp
houqp
opened on Aug 15, 2021

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We had some oversights in the 5.0.0 release (#771) causing us not able to release the python binding and datafusion-cli.

Describe the solution you'd like

Release Datafusion 5.1.0 with an improved process to support python binding and cli releasse.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
see #887, #883 and #837

Activity

  1. houqp
    added
    enhancementNew feature or request
    on Aug 15, 2021
  2. houqp
    self-assigned this
    on Aug 15, 2021
  3. houqp
    added this to the 5.1.0 milestone on Aug 20, 2021

  4. mmuru commented on Sep 3, 2021

    @mmuru
    mmuru
    on Sep 3, 2021
    Contributor

    @houqp: Do we have an ETA on the python binding release? Thanks.

  5. houqp commented on Sep 3, 2021

    @houqp
    houqp
    on Sep 3, 2021
    MemberAuthor

    @mmuru probably 2-3 weeks, need to get some legal issues resolved in #920 before we can cut a release tarball and start the voting process.

  6. mmuru commented on Sep 3, 2021

    @mmuru
    mmuru
    on Sep 3, 2021
    Contributor

    @houqp: Thanks for the update. I am trying to build python wheel locally but I noticed in Cargo.toml file the dependency listed
    datafusion = { git = "https://github.com/apache/arrow-datafusion.git", rev = "4d61196dee8526998aee7e7bb10ea88422e5f9e1" }
    It did not get updated. Will maturin develop pick up the latest datafusion code? Please, can you clarify it? Thanks again.

  7. houqp commented on Sep 4, 2021

    @houqp
    houqp
    on Sep 4, 2021
    MemberAuthor

    @mmuru it will not, I just sent #967 to handle the datafusion update.

  8. mmuru commented on Sep 5, 2021

    @mmuru
    mmuru
    on Sep 5, 2021
    Contributor

    @houqp: Thanks for the clarification and quick turnaround fix. I verified your changes and found two issues.

    1. the requirement.txt is locked to Python 3.8 version. I think, this file should not be checked in the source since the python development version could be different. In my case, it was python 3.7 version.
    2. Fixed the issue Creating dataframe with Recordbatch using pyarrow.Table.to_batches gives "type16 not valid error" when schema includes date32[day] type #949. Added support for pyarrow datatypes such as date32, date64 and Timestamp.
      I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

  9. houqp commented on Sep 5, 2021

    @houqp
    houqp
    on Sep 5, 2021
    MemberAuthor

    the requirement.txt is locked to Python 3.8 version.

    The requirements.txt is supposed to work for all python versions supported by the python binding. if it's broken for python 3.7, could you file a separate issue with the error message? We can continue the troubleshooting there.

    I would like to submit my changes as part of update datafusion to 5.1.0 for python binding #967 PR. Please, let me know.

    That's great, I recommend you send a separate PR based off the branch in #967 for this, or collaborate on #969 to get that issue addressed.

  10. mmuru commented on Sep 6, 2021

    @mmuru
    mmuru
    on Sep 6, 2021
    Contributor

    @houqp: Sure, created #975. Ping me if you need more information.

  11. houqp commented on Sep 30, 2021

    @houqp
    houqp
    on Sep 30, 2021
    MemberAuthor

    @andygrove @alamb @Dandandan @jorgecarleitao @nevi-me given that we have had many major breaking changes merged in since the 5.x release, I am thinking maybe it's better to skip 5.1 and go 6.0 after #1010 gets merged. What do you think?

  12. alamb commented on Sep 30, 2021

    @alamb
    alamb
    on Sep 30, 2021
    Contributor

    merged in since the 5.x release

    I think using 6.0 is a good idea. I also don't think we need to wait for #1010 to be merged for a release, if we need to get the python binding / cli out sooner

  13. houqp commented on Oct 2, 2021

    @houqp
    houqp
    on Oct 2, 2021
    MemberAuthor

    sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

  14. alamb commented on Oct 2, 2021

    @alamb
    alamb
    on Oct 2, 2021
    Contributor

    sounds good, I think think I can try help push #873 to the finish line after you have arrow 6 released.

    It sounds like we are aiming to release arrow 6.0 in 2 weeks or so

  15. houqp
    changed the title [-]Release Datafusion 5.1.0[/-] [+]Release Datafusion 6.0.0[/+] on Oct 9, 2021

  16. 8 remaining items

  17. houqp commented on Nov 17, 2021

    @houqp
    houqp
    on Nov 17, 2021
    MemberAuthor

    @jimexist we should be able to update datafusion-cli in homebrew as well.

  18. jimexist commented on Nov 17, 2021

    @jimexist
    jimexist
    on Nov 17, 2021
    Member

    Homebrew/homebrew-core#89562

  19. alamb commented on Nov 17, 2021

    @alamb
    alamb
    on Nov 17, 2021
    Contributor

    Vote passed and I have pushed the release tags into Github. The release steps requires PMC member access. @alamb @andygrove @jorgecarleitao @kszucs could one of you follow the steps in https://github.com/apache/arrow-datafusion/tree/master/dev/release#finalize-the-release to complete the release?

    @houqp I will do so now. Thank you for all the work in this regard

  20. houqp commented on Nov 17, 2021

    @houqp
    houqp
    on Nov 17, 2021
    MemberAuthor

    Thanks @alamb ! I just noticed I forgot to add (cd datafusion-cli && cargo publish) in the release doc, could you help run that command as well? I will send a PR to get the doc updated later today.

    @andygrove @jorgecarleitao @kou @kszucs @xhochy we will need your help to publish the python binding to PyPI since only you are listed as maintainers of the PyPI package. The steps are documented at https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-pypi

  21. xhochy commented on Nov 17, 2021

    @xhochy
    xhochy
    on Nov 17, 2021
    Member

    I'd rather give more people access to PyPI ;)

  22. kou commented on Nov 17, 2021

    @kou
    kou
    on Nov 17, 2021
    Member

    I'm trying.

    I found a typo in the document:

    diff --git a/dev/release/README.md b/dev/release/README.md
index 2127dc23..73b3eb1a 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -304,7 +304,7 @@ PyPI, in order to conform to Apache Software Foundation governance standards.
 First, download all official python release artifacts:
 
 ```shell
-svn co https://dist.apache.org/repos/dist/release/arrow/apache-arrow-datafusion-5.1.0-rc0/python ./python-artifacts
+svn co https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-5.1.0/python ./python-artifacts
 ```
 
 Use [twine](https://pypi.org/project/twine/) to perform the upload.

    https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-6.0.0/python/ uses 0.4.0 not 6.0.0. Is it OK?

  23. houqp commented on Nov 17, 2021

    @houqp
    houqp
    on Nov 17, 2021
    MemberAuthor

    I'd rather give more people access to PyPI ;)

    +1 :D

    Good catch @kou , I will include that fix in my docs PR. The version diff is expected because we want them to be decoupled so we can release major version change in the python binding without forcing a major version bump in datafusion.

  24. kou commented on Nov 17, 2021

    @kou
    kou
    on Nov 17, 2021
    Member

    OK. I've published them: https://pypi.org/project/datafusion/0.4.0/

    I found one more typo:

    diff --git a/dev/release/README.md b/dev/release/README.md
index 2127dc23..fcf090e3 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -310,7 +310,7 @@ svn co https://dist.apache.org/repos/dist/release/arrow/apache-arrow-datafusion-
 Use [twine](https://pypi.org/project/twine/) to perform the upload.
 
 ```shell
-twine upload ./python-artifactl/*.{tar.gz,whl}
+twine upload ./python-artifacts/*.{tar.gz,whl}
 ```
 
 ### Call the vote

  25. houqp commented on Nov 17, 2021

    @houqp
    houqp
    on Nov 17, 2021
    MemberAuthor

    Thank you @kou ! I will include that fix in my docs PR as well :)

  26. houqp
    closed this as completedon Nov 17, 2021

  27. alamb commented on Nov 18, 2021

    @alamb
    alamb
    on Nov 18, 2021
    Contributor

    Thanks @alamb ! I just noticed I forgot to add (cd datafusion-cli && cargo publish) in the release doc, could you help run that command as well? I will send a PR to get the doc updated later today.

    Hi @houqp

    I tried to publish datafusion-cli and I got the following error. It looks like datafusion-cli relies on ballista somehow

    (arrow_dev) alamb@MacBook-Pro:~/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli$ cargo publish
    Updating crates.io index
warning: manifest has no description.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)
error: failed to prepare local package for uploading

Caused by:
  failed to select a version for the requirement `ballista = "^0.6.0"`
  candidate versions found which didn't match: 0.5.0, 0.3.0, 0.2.5, ...
  location searched: crates.io index
  required by package `datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)`

  28. houqp commented on Nov 18, 2021

    @houqp
    houqp
    on Nov 18, 2021
    MemberAuthor

    oh yeah, it supports ballista as a way to perform remote query execution. @alamb looks like we haven't published the ballista crates yet? could you do that first by following https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-cratesio?

  29. alamb commented on Nov 18, 2021

    @alamb
    alamb
    on Nov 18, 2021
    Contributor

    could you do that first by following https://github.com/apache/arrow-datafusion/tree/master/dev/release#publish-on-cratesio?

    Done (updated instructions in #1331)

    Turns out I still can't upload datafusion-cli package:

    (arrow_dev) alamb@MacBook-Pro:~/Downloads/apache-arrow-datafusion-6.0.0$ (cd datafusion-cli && cargo publish)
....
    Finished dev [unoptimized + debuginfo] target(s) in 1m 20s
   Uploading datafusion-cli v5.1.0-SNAPSHOT (/Users/alamb/Downloads/apache-arrow-datafusion-6.0.0/datafusion-cli)
error: failed to publish to registry at https://crates.io

Caused by:
  the remote server responded with an error: invalid upload request: invalid length 7, expected at most 5 keywords per crate at line 1 column 3441

    I will file a ticket Tracked by #1332

  30. unkloud
    added a commit that references this issue on Mar 23, 2025

    fix: Fix various memory leaks problems (apache#890)

    06bb321
  31. HairstonE
    added a commit that references this issue on Oct 7, 2025

    Feature/instance udfs (apache#890)

    1fd3762
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions