feat(java/driver/jni): implement incremental execution, Substrait

[lidavidm]

Also, fix an incidental bug in the Go driver FFI layer.

Closes #3257.

[Copilot]

Pull request overview

This PR extends the Java JNI ADBC driver to support ADBC 1.1 features (incremental/partition polling, progress reporting) and Substrait plan execution, while also fixing a Go driver FFI bug around Substrait plan memory ownership. It adds/updates FlightSQL integration tests and ensures CI starts the needed FlightSQL test service.

Changes:

• Add Substrait plan support and incremental partition polling/progress APIs to the Java JNI statement implementation.
• Refactor JNI ByteBuffer handling in C++ and add JNI entrypoint for AdbcStatementSetSubstraitPlan.
• Fix Go FFI Substrait plan handling by copying the plan bytes; extend FlightSQL integration tests and CI dependencies.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
java/driver/jni/src/main/java/org/apache/arrow/adbc/driver/jni/JniStatement.java	Implements setSubstraitPlan, pollPartitioned, and progress getters for the JNI statement.
java/driver/jni/src/main/java/org/apache/arrow/adbc/driver/jni/JniDriver.java	Adds typed option keys for incremental execution and progress reporting.
java/driver/jni/src/main/java/org/apache/arrow/adbc/driver/jni/impl/NativeAdbc.java	Declares the new native method for setting a Substrait plan.
java/driver/jni/src/main/java/org/apache/arrow/adbc/driver/jni/impl/JniLoader.java	Wires Java calls through to the new native method.
java/driver/jni/src/main/cpp/jni_wrapper.cc	Adds shared ByteBuffer extraction helper, implements JNI for Substrait plan, and refactors partition read path.
java/driver/jni-validation/src/test/java/org/apache/arrow/adbc/driver/jni/FlightSqlSqliteIntegrationTest.java	Adds FlightSQL/SQLite integration coverage for query + partitioned reads.
java/driver/jni-validation/src/test/java/org/apache/arrow/adbc/driver/jni/FlightSqlIntegrationTest.java	Expands mock FlightSQL integration coverage for Substrait, poll/progress, and cancellation.
go/adbc/pkg/panicdummy/driver.go	Fixes FFI plan lifetime by cloning plan bytes before passing to Go statement.
go/adbc/pkg/flightsql/driver.go	Same FFI plan lifetime fix for FlightSQL Go driver.
go/adbc/pkg/_tmpl/driver.go.tmpl	Propagates the FFI plan lifetime fix into generated drivers.
go/adbc/driver/flightsql/cmd/testserver/main.go	Adds Substrait plan handling paths in the FlightSQL test server.
.github/workflows/java.yml	Starts the flightsql-test service for Java CI runs.

Files not reviewed (2)

• go/adbc/pkg/flightsql/driver.go: Language not supported
• go/adbc/pkg/panicdummy/driver.go: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lidavidm]

CI failures are because GitHub is throttling GitHub.

  Will fetch Miniforge3 latest from https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
  Ensuring Installer...
  Checking for cached Miniforge3@latest...
  Did not find Miniforge3-MacOSX-arm64.sh latest in cache
  Unexpected HTTP response: 504
  Waiting 20 seconds before trying again
  Unexpected HTTP response: 504
  Waiting 11 seconds before trying again
Error: Unexpected HTTP response: 504