Buildable release builds

Problem description

From security standpoint it would be great to have reproducible (byte-for-byte) release builds, however this issue is not about this (but is a prerequisite thereof).
DataFusion downstream consumers consume DataFusion source code, via crates or forking.

Im both these situations it would be great to guarantee that a given release actually builds, not only at the moment of the release.

Observed

A fork of 43.0.0 release no longer passes tests today: https://github.com/findepi/datafusion/actions/runs/13130900468
A fork of 44.0.0 release no longer passes clippy today: https://github.com/findepi/datafusion/actions/runs/13132627770

Expected

A release compiles and works correctly (including passing test). Ideally it also passes clippy.
This is beneficial to anyone consuming a DataFusion code. People running a permanent fork likely mostly figured how to solve this already, but that would be very beneficial to anyone running a particular DataFusion version and wanting to fix a bug. Being able to check out code for the version they use and reproduce the bug there is really good first step, but this requires that the code works.

Proposed solution: commit Cargo.lock to the repository.
Lack of Cargo.lock in the repo gives some benefit (checking with latest versions of unpinned deps automatically). This can be provided with a CI job. I.e. make this a machine problem, not a human (consumer) problem.

Proposed solution: commit Cargo.lock to the repository. Lack of Cargo.lock in the repo gives some benefit (checking with latest versions of unpinned deps automatically). This can be provided with a CI job. I.e. make this a machine problem, not a human (consumer) problem.

#14135

@mbrobbel

As @mbrobbel says, I agree it is time to put Cargo.lock int he repo:

Discuss: Check in Cargo.lock file? #14135

Great. This probably would solve the biggest issue with re-buildable releases.
Some other stuff of similar kind is using pinned vs latest stable rust version.

Some other stuff of similar kind is using pinned vs latest stable rust version.

I think this can be set using a rust-toolchain.toml file, for example https://github.com/influxdata/influxdb/blob/main/rust-toolchain.toml

[toolchain]
channel = "1.84.0"
components = ["rustfmt", "clippy", "rust-analyzer"]

Let's do this. we upgrade compiler version manually anyway, so we can update one more place i believe.

Let's do this. we upgrade compiler version manually anyway, so we can update one more place i believe.

It would help us in InfluxDB too as it we have a patched version of DataFusion we keep around too.

I'll try and make a PR for it over the next few days if no one beats me to it

Here is a proposed PR:

Specify rust toolchain explicitly, document how to change it #14655

Navigation Menu

Repository navigation

Buildable release builds #14479

Problem description

Observed

Expected

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Buildable release builds #14479

Description

Problem description

Observed

Expected

Activity

findepi commented on Feb 4, 2025

mbrobbel commented on Feb 4, 2025

alamb commented on Feb 4, 2025

findepi commented on Feb 4, 2025

alamb commented on Feb 5, 2025

findepi commented on Feb 5, 2025

alamb commented on Feb 7, 2025

alamb commented on Feb 14, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions