feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json by DebadityaHait · Pull Request #2397 · apache/datafusion-ballista

DebadityaHait · 2026-08-30T22:46:54Z

Which issue does this PR close?

Rationale for this change

Ballista's public REST interfaces currently lack a machine-readable OpenAPI specification. As proposed in #2351, this PR addresses Phase 1 by using utoipa to generate OpenAPI documentation directly from our Rust DTO types and serving the spec at GET /api/openapi.json.

What changes are included in this PR?

Added utoipa (5.5) as a workspace dependency with macros enabled.
Added an optional utoipa feature to ballista-api-types and derived utoipa::ToSchema on shared DTO types (JobResponse, TaskStatus, TaskSummary, Percentiles, QueryStageSummary, QueryStagesResponse, PlanFormat).
Added #[utoipa::path(...)] annotations to all public scheduler endpoints in ballista-scheduler (/api/state, /api/version, /api/executors, /api/executor/{executor_id}, /api/jobs, /api/job/{job_id}, /api/job/{job_id}/config, /api/job/{job_id}/stages, /api/job/{job_id}/dot, /api/job/{job_id}/stage/{stage_id}/dot, /api/metrics, and /api/job/{job_id}/dot_svg when graphviz-support is enabled).
Added ballista_scheduler::api::openapi defining the ApiDoc structure and GET /api/openapi.json route handler.
Added comprehensive unit tests in ballista_scheduler and ballista_api_types asserting schema registration, path coverage, serialization, and endpoint response.

Are there any user-facing changes?

The scheduler REST API now exposes GET /api/openapi.json returning the OpenAPI v3 JSON specification. No existing endpoints or serialization schemas were modified.

milenkovicm · 2026-08-31T09:40:47Z

would you mind having a look @avantgardnerio ?

andygrove

Thanks for picking this up. A machine-readable spec for the REST API is a genuinely useful thing to have, and the cfg_attr gating on ballista-api-types is the right call so downstream users of that crate see no change by default.

I left a few inline comments. The one about ExecutorOperatingSystemSpecificationSchema is the one I would like sorted before this lands.

One thing that does not fit on a line in the diff: docs/source/user-guide/scheduler.md has the table of REST endpoints around line 78. Could you add /api/openapi.json to it?

andygrove · 2026-08-31T15:26:40Z

+/// Operating system level specification of an executor.
+#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, utoipa::ToSchema)]
+#[schema(as = ExecutorOperatingSystemSpecification)]
+pub struct ExecutorOperatingSystemSpecificationSchema {


This mirror struct declares three fields, but the real ExecutorOperatingSystemSpecification in ballista/core/src/serde/scheduler/mod.rs has nine. It also carries os_ver_long, physical_cores, num_disks, total_disk_space, total_available_disk_space and open_files_limit, and all of them get serialized.

Since #[schema(value_type = ...)] swaps the schema out wholesale, the published spec will under-describe what /api/executors and /api/executor/{executor_id} actually return, right from the first release. ExecutorSpecificationSchema above happens to match today, but it is the same hand-copied pattern and will drift the same way as soon as someone edits the core type.

Would it work to derive ToSchema on the real types in ballista-core behind an optional feature, the same way you have done it for ballista-api-types? That way there is only one definition to keep right.

If you would rather keep this PR small, filling in the six missing fields would fix the immediate problem. In that case it might be worth adding a test that asserts the schema's property set matches the JSON keys of ExecutorOperatingSystemSpecification::default(), so the next drift shows up as a failing test rather than a quietly wrong spec.

andygrove · 2026-08-31T15:26:40Z

+struct GraphvizApiDoc;
+
+/// Generate the OpenAPI specification for the scheduler REST API.
+pub fn openapi_spec() -> utoipa::openapi::OpenApi {


This rebuilds the whole document on every request to /api/openapi.json. The spec is fixed at compile time, so wrapping it in a std::sync::LazyLock would turn the handler into a cheap clone of a cached value.

andygrove · 2026-08-31T15:26:40Z

+        crate::api::handlers::get_job_dot_graph,
+        crate::api::handlers::get_query_stage_dot_graph,
+        crate::api::handlers::get_scheduler_metrics,
+        crate::api::openapi::get_openapi_spec,


/healthz and /readyz are missing here. They are mounted by health_routes unconditionally, so they are live even when rest-api is off, and they are exactly the two endpoints a platform owner wires into Kubernetes probes.

Anyone generating a client or a monitoring config from openapi.json will not see them. Happy for this to be a follow-up if you would rather keep Phase 1 focused.

andygrove · 2026-08-31T15:26:40Z

+    use utoipa::ToSchema;
+
+    #[test]
+    fn test_dto_schemas() {


This one is mostly asserting that the derive macro does what it says, so I am not sure it earns its keep. A check that the generated schema's properties line up with the actual serde output would catch a lot more, things like a skip_serializing_if that the schema does not know about.

Worth knowing too that this only runs when feature unification turns the utoipa feature on from the scheduler side. cargo test -p ballista-api-types on its own will skip it silently.

DebadityaHait · 2026-08-31T16:07:58Z

Thank you for the review and suggestions @andygrove! I've addressed all feedback in the latest commit:

Direct ToSchema derivation in ballista-core:
- Added optional utoipa feature to ballista-core and derived ToSchema directly on ExecutorSpecification and ExecutorOperatingSystemSpecification (including all 9 fields).
- Removed the proxy mirror structs in handlers.rs.
- Added property alignment tests comparing utoipa::ToSchema property sets directly against serde_json serialization output to guard against future schema drift.
Cached OpenAPI Document:
- Wrapped the spec assembly in std::sync::LazyLock so GET /api/openapi.json returns a cached value.
Kubernetes Health Probes (/healthz and /readyz):
- Added #[utoipa::path] annotations to healthz and readyz in api::health and included them in the OpenAPI specification under the health tag.
Schema Property Alignment Tests:
- Enhanced tests in both ballista-api-types and ballista-scheduler to verify that generated schema properties match the serialized JSON keys of DTO and response instances.
Documentation:
- Added GET /api/openapi.json to the REST endpoints table in docs/source/user-guide/scheduler.md.

avantgardnerio

I see you addressed @andygrove 's feedback, so ✔️ with one nice-to-have in the comments.

milenkovicm · 2026-09-12T14:51:24Z

Can you please rebase pr @DebadityaHait

…i.json (apache#2351)

…derives, and health probes

DebadityaHait · 2026-09-12T17:00:27Z

Rebased onto latest main, updated Cargo.lock, and verified that all tests, --locked cargo check, and machete checks pass. Thank you @milenkovicm!

milenkovicm · 2026-09-12T20:39:21Z

thanks @DebadityaHait, @avantgardnerio & @andygrove will merge this on green ci

Merging main brought in the OpenAPI endpoint (apache#2397) and build-side staging for AQE joins (apache#2434). Neither was reflected in the docs this branch had already made complete, and auditing them turned up two older inaccuracies in the same sections. From the merged changes: - Document the new `utoipa` feature on ballista-core in the root and crate READMEs, and note that the scheduler's `rest-api` feature now also serves the OpenAPI spec. - Add build-side staging to the AQE optimization lists in the tuning and architecture guides, and to the 55.0.0 upgrade guide, since it is a new default that changes query plans. Pre-existing inaccuracies: - The tuning guide's Join Strategy section only described the static planner. With AQE on by default, `prefer_hash_join` is not consulted; the join is chosen at runtime from the broadcast threshold and `hash_join_max_build_partition_bytes`. The section now covers both paths, and notes that the static planner only promotes hash joins to broadcast. - Join reordering compares byte sizes first and falls back to row counts, not row counts alone. - The REST API was described as an optional feature to enable, but `rest-api` is on by default.

* docs: correct stale defaults, flags, and feature docs AQE was documented as experimental and off by default; it is on by default. The scheduler policy default is push-staged and the event-loop buffer is 1000. ballista-cli has no spark-compat feature. The REST table was missing endpoints and both health probes, the Prometheus list named a metric that is not exported, and every second-executor example moved only --bind-port. CLI usage is regenerated from the built binary. Also brings the docs up to date with #2397 (utoipa, rest-api default) and #2434 (build-side staging). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: keep the Python SessionConfig limitation and name AQE join selection consistently A SessionConfig passed as config is still used only by the local context; only cluster_config reaches the cluster, so restore the limitation and point at cluster_config. The tuning guide listed join reordering and broadcast join selection as two optimizations, while the architecture guide calls them one. Both are the same AQE rule, so merge them into a single Join selection bullet, and say that the build side is chosen by byte size, falling back to row count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: fix the Spark-functions CLI claim, sample logs, and benchmark formats - ballista-cli does not get Spark functions from the cluster: SQL is planned in the client, so the CLI needs them compiled in. Say so, and give the source build that enables them (ballista-core's spark-compat, unified in). - Replace sample startup logs with what the current binaries print. The executor sample showed messages that no longer exist, and a pull-mode poll loop although push-staged is the default. - Keep --format tbl in the benchmark commands; tpch reads dbgen's .tbl files. - Note that the published images lack keda-scaler, that --disable-rest-api turns the REST API off at runtime, and tidy the Prometheus note. - Stop quoting rust_clippy.sh's exact command, which #2473 changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: move Who uses Ballista from the README to the docs The README is the project's entry point, so keep the adopters list in the published docs instead, as a Community page. All three of its links were checked and resolve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

milenkovicm requested a review from avantgardnerio August 31, 2026 09:39

DebadityaHait force-pushed the feat/scheduler-openapi-spec branch from e4dd39c to 243937f Compare August 31, 2026 12:37

andygrove reviewed Aug 31, 2026

View reviewed changes

github-actions Bot added the documentation Improvements or additions to documentation label Aug 31, 2026

DebadityaHait force-pushed the feat/scheduler-openapi-spec branch from 63fd4b2 to db36492 Compare August 31, 2026 16:48

avantgardnerio approved these changes Aug 31, 2026

View reviewed changes

Comment thread ballista/api-types/Cargo.toml

DebadityaHait force-pushed the feat/scheduler-openapi-spec branch from db36492 to 19c79f7 Compare September 1, 2026 20:29

DebadityaHait added 3 commits September 12, 2026 22:05

feat(scheduler): document REST API with OpenAPI and serve /api/openap…

a420bc2

…i.json (apache#2351)

fix(scheduler): address review feedback on OpenAPI spec, core schema …

1cbf9af

…derives, and health probes

test(api-types): enable utoipa feature for tests via self dev-dependency

87c8cd4

DebadityaHait force-pushed the feat/scheduler-openapi-spec branch from 19c79f7 to 87c8cd4 Compare September 12, 2026 16:59

milenkovicm merged commit ae9bff0 into apache:main Sep 12, 2026
27 checks passed

NoahKusaba mentioned this pull request Sep 13, 2026

chore: fix stale docs and CI checks that were not running #2438

Closed

NoahKusaba mentioned this pull request Sep 22, 2026

docs: correct stale defaults, flags, and feature docs #2478

Merged

andygrove mentioned this pull request Sep 29, 2026

Discussion: release the Python client separately from the Rust crates #2511

Open

Navigation Menu

feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json - #2397

feat(scheduler): document REST API with OpenAPI and serve /api/openapi.json#2397
milenkovicm merged 3 commits into
apache:mainfrom
DebadityaHait:feat/scheduler-openapi-spec

DebadityaHait commented Aug 30, 2026 •

edited

Loading

Uh oh!

milenkovicm commented Aug 31, 2026

Uh oh!

andygrove left a comment

Uh oh!

andygrove Aug 31, 2026

Uh oh!

andygrove Aug 31, 2026

Uh oh!

andygrove Aug 31, 2026

Uh oh!

andygrove Aug 31, 2026

Uh oh!

DebadityaHait commented Aug 31, 2026 •

edited

Loading

Uh oh!

avantgardnerio left a comment

Uh oh!

Uh oh!

milenkovicm commented Sep 12, 2026

Uh oh!

DebadityaHait commented Sep 12, 2026

Uh oh!

milenkovicm commented Sep 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

DebadityaHait commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

milenkovicm commented Aug 31, 2026

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Aug 31, 2026

Choose a reason for hiding this comment

Uh oh!

DebadityaHait commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avantgardnerio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

milenkovicm commented Sep 12, 2026

Uh oh!

DebadityaHait commented Sep 12, 2026

Uh oh!

milenkovicm commented Sep 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DebadityaHait commented Aug 30, 2026 •

edited

Loading

DebadityaHait commented Aug 31, 2026 •

edited

Loading