Skip to content

fix(api,event,gateway): preserve encoded upstream paths in gateway routes - #837

Merged
ron96g merged 5 commits into
mainfrom
fix/preserve-encoded-upstream-path
Nov 18, 2026
Merged

ron96g merged 5 commits into
mainfrom
fix/preserve-encoded-upstream-path

Conversation

@nb-99

@nb-99 nb-99 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Intent

Gateway Routes must preserve percent-encoded characters in upstream paths. Before this change, an upstream such as https://provider.example.com/api/url%20encoded%20path produced a Route upstream path with raw spaces. The gateway then could not construct the provider URL and returned HTTP 500. Other encodings were silently changed: %2F became an extra path segment and %2520 lost one encoding layer.

Approach

All modules now convert an upstream URL with one shared helper, gatewayv1.ParseUpstream in gateway/api/v1. It stores url.URL.EscapedPath() instead of the decoded url.URL.Path. EscapedPath() preserves canonical encodings such as %20. RawPath alone is insufficient because Go can leave it empty for these encodings. The Upstream.Path field documentation now states that the path is percent-encoded.

The helper replaces these per-module copies:

Module Upstream source Removed copy
api API exposures, failover, proxy, and remote API subscription consumer routes AsUpstream
agentic Agentic exposures and proxy routes AsUpstream
admin Managed zone routes inline parsing
event Publish, SSE, Voyager, and proxy routes parseUpstream
spectre SSE and SSE proxy routes parseSSEUpstream
file File API service route inline parsing

The Kong client URL parser in gateway/pkg/kong/client also uses EscapedPath().

Re-encoding only in the gateway operator was considered. It cannot restore %2F, because a decoded / is indistinguishable from a path separator.

The user documentation now states that encoded upstream path characters are forwarded unchanged.

Commits

  1. fix: preserve encoded upstream paths in gateway routes: the fix and the regression tests in each module.
  2. refactor: share upstream URL parsing in gateway api: the shared helper, as suggested in review.
  3. test(gateway): run gateway/api tests in make test: gateway/api gets a Makefile, and the gateway module's make test runs it, like approval does for approval/api.
  4. fix(event): query-encode subscriber URL in gateway callback URL: the subscriber URL is added to the gateway callback URL with url.Values, as Spectre already does. Before, a subscriber URL with its own query string, such as ?a=1&b=2, lost every parameter after the first &. The gateway decodes the callback query parameter before it uses it as upstream.

Verification

  • gateway/api/v1/util_test.go covers the helper: %20, %2F, %25, %2520, a plain path, default ports, and parse errors. With EscapedPath() replaced by Path, the four encoding cases fail.
  • Route tests in each module check that the stored Route keeps the encoded path. Before the fix, they failed with the decoded path, for example /api/url encoded path.
  • make test passed in api, agentic, admin, event, file, spectre, gateway, and gateway/api.
  • Event callback: the handler test now uses a subscriber URL with a query string and expects the whole URL in one encoded callback parameter. Before the fix, this test and the encoded expectations in the handler and callback lifecycle tests failed. An end-to-end delivery through the gateway was not tested.
  • make lint reports no findings in changed files. It fails in api, agentic, gateway, and spectre because of findings in unchanged files.

The docs site build was not run.

Reviewer guidance

Start with gateway/api/v1/util.go. The module changes replace local parsing with ParseUpstream. Callers that need a weight set it after the call.

Existing Routes keep their decoded paths until their controllers reconcile them after deployment.

Rollout impact of commit 4: every existing callback subscription gets a new callback URL on its next reconcile, for example ?callback=https://… becomes ?callback=https%3A%2F%2F…. Pubsub then updates each Horizon subscription once. The delivered events go to the same subscriber URL.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The consumer controller test lacks an encoded GatewayUrl case and a persisted upstream-path assertion, leaving the Route mapping vulnerable to undetected regressions.

1 open issue
What changed in this PR

This PR fixes upstream URL mapping in the API controller so gateway Routes preserve percent-encoded paths, preventing provider URL construction failures.

Changes:

  • Use url.URL.EscapedPath() when mapping upstream URLs.
  • Reuse the corrected mapper for remote API consumer Routes.
  • Add mapper and persisted-Route regression tests, and document encoded-path forwarding.
File Description
docs/​docs/​user-journey/​exposing-apis.mdx Documents preservation of encoded upstream paths.
api/​internal/​handler/​util/​gateway.go Preserves path encoding with EscapedPath().
api/​internal/​handler/​util/​gateway_test.go Tests URL components and several encoding cases.
api/​internal/​handler/​util_real_route_test.go Verifies encoded paths survive Route persistence.
api/​internal/​handler/​remoteapisubscription/​consumer_scenario.go Uses the shared upstream mapper for consumer Routes.

Review effort: Max

@nb-99
nb-99 force-pushed the fix/preserve-encoded-upstream-path branch from 5e158fc to 75dd2cd Compare November 6, 2026 13:19
@ron96g

ron96g commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

I would suggest to use some central function for this which the gateway-api module. Alternative could also be that this encoding logic is inside of the gateway-operator only. So no CP changes.

Related but out-of-scope I guess: How is this done for event-callback-urls?

@nb-99

nb-99 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@ron96g I agree with the central function, I'll move it into a shared helper in gateway and have all modules use it. I'd rather not do it only in the operator, once the path is decoded, %2F can't be restored and a path separator cannot be distinguished from an encoded slash.

The event callback query param isn't encoded in eventsubscription/handler.go (spectre does it right). Separate issue or should this PR cover that too?

@nb-99

nb-99 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

With the new helper in gateway/api, we have a new test suite. For now, the gateway Makefile runs it from its test target ($(MAKE) -C api test), the same way approval runs approval/api. So they run in CI whenever something under gateway/ changes.

I didn't find a standard approach for this. Most nested API modules with tests (api/api, rover/api, event/api, admin/api, organization/api) aren't tested in CI at all, and none has an entry in .github/ci/modules.yaml.

@nb-99
nb-99 marked this pull request as ready for review November 12, 2026 13:52
@ron96g

ron96g commented Nov 13, 2026

Copy link
Copy Markdown
Member

AI-assisted investigation: URL-encoding compatibility

This comment documents a local investigation performed by an AI assistant using Copilot SDK in VS Code. Reviewed PR head: 2b61e2f789fde6ed17aa7a03dbe7082c3e8d1701.

Result: no encoding-related downstream incompatibility found in the tested scenarios.

Fully encoded callback URL

The fully encoded subscriber URL is the value of the gateway callback query parameter, not the destination string passed directly to Jumper:

callback=https%3A%2F%2Fmy-callback.example.com%2Fevents%3Ftenant%3Da%26mode%3Db

Kong parses that query parameter and the request-transformer expression $(query_params.callback) sets:

remote_api_url: https://my-callback.example.com/events?tenant=a&mode=b

The outer callback parameter is removed before forwarding to Jumper. This decoding step was verified against a running Kong by capturing its actual outgoing header. Delivery through Jumper then reached the subscriber with both query parameters intact, including through a two-gateway-hop forwarding path. Escapes belonging to the subscriber URL, such as %2F and %2520, survived without additional decoding or double-encoding.

Runtime setup and verification

  • Used temporary local Docker containers: official kong:3.9 (resolved to Kong 3.9.2) in DB-less mode, and Jumper 4.6.1.
  • Used Apache HttpClient 4.5.14 with the request construction used by Horizon Comet and Polaris to exercise callback POST delivery and HEAD/GET health-check requests. The complete Horizon services were not started.
  • The cached Telekom Kong image mtr.devops.telekom.de/tardis-internal/gateway/kong:1.6.1 (Kong 3.9.1) required Admin API authentication, so HTTP tests used stock Kong instead. Separately verified that its request-transformer implementation and relevant request/query-handling Lua files were byte-for-byte identical to those in the tested stock image.
  • 244 assertions passed, checking actual destination request paths, queries, methods, and callback-derived headers. Covered passthrough, single upstream, load-balancing and failover URL forwarding, and one-/two-hop callbacks.
  • Cases included %20, %2F, %25, %2520, lowercase escapes, UTF-8, encoded delimiters, and queries with multiple parameters, literal +, %2B, repeated keys, and empty values.
  • Negative controls reproduced the old HTTP 500 for decoded spaces, changed path segmentation for decoded slashes, and callback-query truncation.
  • Targeted Makefile tests passed for gateway parser/features/Kong client, event callback/controller/route logic, and pubsub subscriber/service logic.

Scope and rollout notes

This was local runtime/HTTP verification, not a full deployed Kafka/Horizon/identity integration test or a test of the deployed gateway version. No deployed/shared gateway instance was used; temporary containers were stopped and removed. No repository code changes were made.

Existing encoded upstreams will change to their intended semantics when reconciled: for example, %2F remains an encoded slash rather than becoming a path separator. Existing callback subscriptions receive a one-time URL update on reconciliation.

Recommendation: retain downstream wire-level regression coverage, since parser/configuration assertions alone cannot establish consumer compatibility.

The subscriber callback URL was appended to the gateway callback URL
without encoding. A subscriber URL with its own query string, such as
"?a=1&b=2", lost every parameter after the first "&" because the
gateway read them as separate parameters.

Build the query with url.Values, as Spectre already does. The gateway
decodes the "callback" query parameter before it uses it as upstream.

Existing callback subscriptions get the encoded URL on their next
reconcile, which updates their Horizon subscription once.
@nb-99
nb-99 force-pushed the fix/preserve-encoded-upstream-path branch from 2b61e2f to 1a1381c Compare November 13, 2026 10:51
ron96g
ron96g previously approved these changes Nov 18, 2026
Resolve conflicts with #864 (callback query param renamed to "url") by keeping the query-encoded subscriber URL under the new "url" parameter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ron96g ron96g changed the title fix(api): preserve encoded upstream paths in gateway routes fix(api,event,gateway): preserve encoded upstream paths in gateway routes Nov 18, 2026
@ron96g
ron96g merged commit 551f3be into main Nov 18, 2026
65 checks passed
@ron96g
ron96g deleted the fix/preserve-encoded-upstream-path branch November 18, 2026 11:28
dtit-semantic-release Bot pushed a commit that referenced this pull request Nov 18, 2026
## [0.48.3](v0.48.2...v0.48.3) (2026-11-18)

### Bug Fixes

* **api,event,gateway:** preserve encoded upstream paths in gateway routes ([#837](#837)) ([551f3be](551f3be))
* **discovery-server:** expose icto and psiid of applications ([#858](#858)) ([1237c46](1237c46))
* **rover,event:** additional publishers get a client ([#872](#872)) ([af3d8e2](af3d8e2))

### Performance Improvements

* **rover-server:** stop resolving secrets when listing rovers ([#870](#870)) ([50a5f62](50a5f62))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants