perf(gateway): reverse proxy throughput optimizations#825
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
kvinwang
force-pushed
the
gateway-perf-clean
branch
from
July 26, 2026 08:34
84c0238 to
8be5386
Compare
kvinwang
marked this pull request as ready for review
July 27, 2026 05:49
With TCP_NODELAY enabled, an 8 KiB copy buffer emits many small tail segments and caps bulk throughput. A 64 KiB buffer keeps writes at full segment size; measured ~2x passthrough/terminate throughput on a 4-core gateway with no latency regression. 128 KiB adds <5% for double the per-connection memory, so 64 KiB is the chosen knee.
TLS 1.2 resumption already works via the in-memory session-ID cache, but TLS 1.3 resumption needs a ticketer. Without one, every TLS 1.3 reconnect pays a full handshake (RSA-2048 signing). Measured on tdxlab (4-core gateway, wrk Connection: close): TLS 1.3 reconnect CPS 6.2k -> 9.6k (+53%) with the ticketer, and TLS 1.2 CPS unchanged (~20k). Default TLS version is unchanged.
The accept/handle hot path logged three info-level events plus an info-level span per connection. At the default info log level this cost ~4-8% of TLS-terminate CPS (measured ~19.2k vs ~20.8k req/s on a 4-core gateway). Downgrading them to debug (and the span to debug_span) removes the per-connection logging cost at info while keeping the detail available when debug logging is enabled.
The TLS-passthrough path is a pure TCP relay (the gateway never sees plaintext), so it can move bytes kernel-side with splice(2) through a pipe instead of copying through a userspace buffer. Gated behind the new proxy.tcp_splice_enabled flag (default off, Linux only). Measured on a 4-core gateway (tdxlab): bulk passthrough throughput 7.60 -> 8.52 GB/s (+12%) with lower tail latency under load (p99 38 -> 21 ms). Small-request passthrough latency regresses (extra pipe hop per tiny message), so it stays opt-in. Data integrity verified with a 100 MiB checksum round-trip.
Adds proxy.ktls_enabled (default off, Linux only). The handshake still runs in rustls; afterwards the negotiated keys are installed into the kernel TLS ULP so record crypto happens there, and when combined with tcp_splice_enabled the payload is relayed with splice and never enters userspace. Measured on a dedicated 44-vCPU GCP VM with the gateway pinned to 4 cores (isolated A/B, interleaved rounds): rustls (userspace) 32226 CPS 7.69 GB/s kTLS alone 20598 CPS 5.72 GB/s (-36% / -26%) kTLS + splice 22640 CPS 9.63 GB/s (-30% / +25%) kTLS alone is a loss: without NIC TLS offload (TlsTxDevice = 0) the kernel's software AES-GCM is slower than rustls with aws-lc-rs, plus per-connection setup cost. Combined with splice it gives the highest terminate throughput measured, so it is worth having for bulk-transfer deployments -- but it stays off by default because connection setup rate drops ~30%. Security note: enabling this extracts session keys into the kernel via dangerous_extract_secrets. Inside a CVM the kernel is part of the measured TCB; on a normal host this widens key exposure. Hence opt-in. Verified with a 100 MiB checksum round-trip and TlsDecryptError = 0.
kTLS' cost and benefit sit on opposite sides of the connection-reuse axis: setup is paid per connection (~34% of connection rate) while the win is per byte (+29% bulk throughput). Offloading at handshake time therefore penalises short request/response connections that never earn it back. With the new proxy.ktls_offload_after_bytes, a connection starts in userspace rustls and is handed to the kernel only once it has moved that many bytes: flush, let CorkStream stop at a TLS record boundary, extract the current traffic secrets (they carry the record sequence numbers, which is what makes a mid-stream handover sound), install them into the kernel, splice the rest. 0 keeps the previous immediate-offload behaviour. Measured on a dedicated 44-vCPU GCP VM, gateway pinned to 4 cores, threshold 1 MiB (2 interleaved rounds): rustls 32170 CPS 7.70 GB/s 164459 small-req/s kTLS immediate 21212 CPS 9.95 GB/s 193208 small-req/s (-34% CPS) kTLS adaptive 26303 CPS 9.93 GB/s 194720 small-req/s (-18% CPS) So it recovers about half the lost connection rate while keeping the full throughput gain, and long-lived small-request connections gain ~18% at lower CPU because they cross the threshold and then run zero-copy. Verified: with a 1 MiB threshold a small request does not offload (kernel TlsTxSw unchanged) while a 100 MiB transfer does (counter +1), checksum matches and TlsDecryptError stays 0.
Profiling the small-request workload showed the gateway leaving ~0.5 of 4
cores idle while HAProxy pinned all four. The cause was scheduling, not the
data path: the gateway performed ~0.617 context switches per request against
HAProxy's ~0.000. Two sources, both structural:
* accept runs on its own current_thread runtime and hands every connection
to a separate multi-threaded worker runtime -- one cross-runtime wakeup
per connection;
* tokio's work-stealing scheduler then migrates those tasks between worker
threads, costing further wakeups and cache locality.
With proxy.thread_per_core = true the proxy instead runs `workers` threads,
each with its own current_thread runtime and its own SO_REUSEPORT listener.
The kernel load-balances new connections across the listeners and a connection
is accepted and served entirely on one thread, so nothing is handed off.
Measured on a dedicated 44-vCPU GCP VM, 4 cores, vs the default model and
vs HAProxy given matching settings:
default thread-per-core HAProxy
terminate small-req 158k @3.55 267k @4.03 204k @4.03
terminate CPS 32.8k @3.65 45.0k @3.96 28.2k @4.00
passthrough small-req 175k @3.56 312k @4.02 355k @4.02
terminate throughput 7.5 GB/s 8.3 GB/s 6.0 GB/s
Context switches per request drop to ~0.000 and the proxy now saturates its
cores. That turns two of the three deficits against HAProxy into wins
(terminate small-request +31%, terminate CPS +60%) and closes most of the
third (passthrough small-request now -12%, was -63%).
Off by default: it changes the process model, needs SO_REUSEPORT (Linux), and
per-core accept queues mean a slow connection can only be served by the core
that accepted it. Verified with 100 MiB checksum round-trips on both paths.
Creating a pipe per direction per connection costs two pipe2 calls plus four descriptor closes. That is pure overhead for short connections and it showed: enabling splice raised passthrough connection-setup CPU from 134 to 142 us per connection, while HAProxy -- which pools its pipes -- moved the other way. Pipes now come from a thread-local pool (cap 64), and are only returned once the transfer has drained them; a pipe with unknown residue is closed rather than risk feeding stale bytes to the next connection. Thread-local means no locking, and under thread_per_core a connection stays on the thread that borrowed the pipe. Measured on the 4-core GCP bed (passthrough, 2 interleaved rounds): small-request rate 317.8k -> 335.9k rps (+5.7%, HAProxy 336.8k) bulk throughput unchanged at ~13 GB/s connection setup unchanged So the win is on keep-alive small-request traffic, where it brings the gateway level with HAProxy. Verified with repeated 100 MiB checksum round-trips and interleaved small/large requests, which is what would expose a dirty pipe.
connect_multiple_hosts always built a JoinSet and spawned a task per candidate address so it could race them. With a single candidate -- the common case for an app with one instance -- there is nothing to race, and the allocation plus task spawn was paid on every connection. Take the address directly in that case. Measured on the 4-core GCP bed (passthrough connection setup, 2 interleaved rounds): 139.7 -> 130.7 us of CPU per connection (-6.4%), narrowing the gap to HAProxy's 123.1 us from 13.5% to 6.2%.
The pool only recycled a pipe when its direction observed EOF, so the other direction's pipe was closed and recreated on every connection -- syscall counting showed a steady 1.00 pipe2 per connection instead of ~0. The pipe is in fact empty at every point between chunks, because each chunk is fully drained before the next splice from the socket. Track that directly: mark the pipe recyclable after each complete drain, and only un-mark it while a chunk is in flight. A pipe is still discarded rather than pooled if the connection dies mid-drain. Measured on the 4-core GCP bed (passthrough): pipe2 per connection 1.00 -> 0.00 connection-setup CPU 132.4 -> 120.1 us median (HAProxy 123.4) That closes the last connection-setup deficit against HAProxy; the two now overlap within run-to-run spread. Re-verified with repeated 100 MiB checksum round-trips and interleaved small/large requests, which is what a dirty pooled pipe would corrupt.
splice costs ~17 syscalls per connection to move a small response -- fill the
pipe, drain the pipe, plus readiness retries -- where a read/write pair needs
two. Its benefit is per byte but its cost is per connection, the same shape as
kTLS, so short request/response connections paid for zero-copy they never used.
With the new proxy.tcp_splice_after_bytes a passthrough connection is relayed
with plain reads/writes until it has moved that many bytes, then switches to
splice for the remainder. 0 keeps the previous behaviour.
The relay buffers come from a thread-local pool for the same reason the pipes
do: a first cut that allocated two buffer_size buffers per connection was
*worse* than the pipe setup it replaced (148.6 us vs 133.5 us per connection).
Pooled, and sized for this phase rather than for bulk copying, it wins.
Measured on the 4-core GCP bed (passthrough, medians of 3 interleaved rounds,
threshold 64 KiB):
always splice adaptive HAProxy
connection setup 123.0 us 117.2 us 123.5 us
small-request rate 300.0k rps 295.3k rps 344.6k rps
bulk throughput 12.6 GB/s 12.4 GB/s 10.2 GB/s
splice syscalls per connection drop from 16.86 to 0.00 on the connection-churn
workload, and connection setup costs slightly less CPU than HAProxy. Keep-alive
and bulk traffic are unchanged within run-to-run spread (~2%), since those
connections cross the threshold and end up spliced anyway.
Verified with checksum round-trips at 1 MiB and 100 MiB plus payloads sized
just under and just over the threshold, which is where the handover happens.
The thread-per-core path built its listener with socket2 and a 4096-deep accept queue; the default path used TcpListener::bind, i.e. tokio's default backlog of 1024. That is where SYN drops begin under connection bursts, and it meant the two modes behaved differently for no reason. Both paths now share one socket2-based bind, differing only in whether SO_REUSEPORT is set, with the backlog in a named constant.
A 40-minute soak produced 379 error-level lines and not one real fault. They were all the peer hanging up: connection reset while bridging, while splicing, or during the TLS handshake. That is routine -- a browser navigating away, a mobile network dropping, a load generator ending its run -- and logging it at error level buries the failures that are actually the gateway's fault. Classify the error chain: if any cause is a ConnectionReset, BrokenPipe, UnexpectedEof or ConnectionAborted io::Error, log at debug instead. Also records what the soak found about the splice pipe pool's descriptor cost (PIPE_POOL_MAX * 2 * workers, so ~512 for 4 workers and ~4096 for 32), which is why that cap is not larger.
… probe Thread-per-core measured 22-77% better on every metric, on loopback and again over a real network, and it is what turned the deficits against HAProxy into wins. The only thing holding it back was that it had never run longer than a benchmark. A soak of 8 x 60s of mixed load (small-request, bulk, connection-churn) says it is stable: RSS 21.9 MB cold -> ~44 MB steady, flat across all rounds fds 31 -> 543, plateauing at the pipe pool cap, then flat threads 11, constant drift small-request -2.3%, connection rate -1.3% (both within noise) Flipping the default needs a safety net, because the model cannot work without SO_REUSEPORT and failing to serve is much worse than losing the optimisation. `start` now probes a SO_REUSEPORT bind first and falls back to the shared-runtime proxy with a warning if it fails. Verified by occupying the port with a non-REUSEPORT socket: the gateway logs the fallback and keeps serving. Deployments that want the old process model can still set proxy.thread_per_core = false. The one behavioural difference to be aware of is that per-core accept queues mean a connection is served by the core that accepted it, so a slow core cannot be helped by its neighbours.
…bridge
Two costs the small-request passthrough profile put at the top of userspace,
both in io_bridge:
* `tokio::io::split` wraps the stream in a `BiLock`, and its `poll_read`
showed at 1.1% of total time. Passthrough always has a plain socket on both
sides, so it can use `TcpStream::split`, which hands out borrowed halves
with no lock. Added `bridge_tcp` for that case.
* every read, write and flush was wrapped in `tokio::time::timeout`, i.e.
three timer arm/disarm pairs per request; `TimerEntry` drop plus
poll_elapsed came to 2.9%. Replaced with one watchdog per connection that
samples per-direction progress counters -- the hot path now costs an integer
increment instead of touching the timer wheel, and no clock is read.
The watchdog is coarser than the old per-operation timeouts: it fires within
`idle/4`, so a stalled connection is closed in up to 1.25x `idle` rather than
exactly at it. Verified with `idle = 3s`: old path closed at 3.0s, new at 3.7s.
Measured on the 4-core GCP bed with the corrected harness (physical-core
disjoint pinning, warmed up, n=5, sd<1%):
passthrough small-request, default config 257 151 -> 290 440 rps (+12.9%)
cpu per request 15.7 us -> 13.9 us
p99 221 us -> 197 us
Note this only moves the default path: with `tcp_splice_enabled` the relay is
`splice_bidirectional`, which never used either mechanism.
An earlier attempt to answer the same question by toggling
`data_timeout_enabled` reported "no effect" -- that was wrong, because the bench
config generator wrote the key under `[core.proxy]` instead of
`[core.proxy.timeouts]`, so serde silently ignored it and both arms ran with
timers on.
`splice_one` awaited `readable()` / `writable()` before every splice, even when the socket was already known to be ready. That builds, polls and drops a `Readiness` future each time; on the small-request passthrough profile `ScheduledIo::poll_readiness` plus `Readiness::drop` came to 2.4% of total time. tokio's `try_io` already decides with a single atomic read of the cached readiness flag and returns WouldBlock without calling the closure if the socket is not ready, so the wait is only needed on that path. Reordered to attempt the syscall first and await readiness only when it blocks. Measured (4-core GCP bed, corrected harness, n=5, sd 0.3%): passthrough small-request, splice enabled 291 546 -> 294 583 rps (+1.0%) cpu per request 13.9 us -> 13.7 us Smaller than the profile share suggested, but reproducible and above noise. 100 MiB checksum round-trip re-verified.
Each thread-per-core worker used to bind its own SO_REUSEPORT listener, so the order sockets joined the group was whatever the thread scheduler produced. Bind them all up front instead and hand listener i to worker i: the setup leaves the per-thread path, and group order becomes ours. That order matters for anything that steers connections by listener index, which is what motivated this. Recording the result of that attempt here, because it was a dead end worth not repeating: SO_REUSEPORT's default 4-tuple hash distributes moderate numbers of long-lived connections unevenly, and thread-per-core cannot rebalance afterwards. At 16 connections over 4 cores the per-core utilisation came out like [99, 42, 101, 101] and throughput swung 46% between restarts purely on how the hash fell. SO_ATTACH_REUSEPORT_CBPF was tried to replace the hash. Classic BPF has no maps, so round-robin is not expressible; the only useful signal it exposes is the CPU handling the packet, so the program steered by `cpu % n`. Measured over 6 restarts each at 16 connections: default hash 245 539 rps 4.0 of 4 cores active lowest core 73% cpu steering 167 331 rps 2.3 of 4 cores active lowest core 100% It collapses the distribution instead of evening it out: connections are established in a burst, so only the one or two client CPUs live at that moment get sampled, and every connection lands on the same one or two listeners. The knob was removed rather than shipped -- a setting that costs 32% in the only scenario measured is worse than no setting. A real fix needs per-connection state, i.e. SO_ATTACH_REUSEPORT_EBPF with a counter map, or accepting from a shared queue as HAProxy does.
SO_REUSEPORT picks a listener by hashing the connection's 4-tuple, and thread-per-core cannot move a connection afterwards, so a core that draws few connections idles while its neighbours saturate. At 16 connections over 4 cores utilisation came out like [99, 42, 101, 101] and throughput varied 46% between restarts on nothing but how the hash fell. Steering the kernel's choice with SO_ATTACH_REUSEPORT_CBPF was tried first and measured worse (see the previous commit). This fixes it on our side instead: the accepting core compares its own connection count against the least loaded core and, if it is ahead by more than a small slack, sends the connection over a channel for that core to serve. The cost is one channel send per *rebalanced connection* -- never per connection, never per request -- so everything already in flight keeps the thread-per-core locality that made this model fast. Measured over 6 restarts per arm, 4-core gateway, passthrough small-request: connections off on 16 236 110 rps, worst core 72% 251 840 rps, worst core 85% (+6.7%) 50 291 932 rps 287 440 rps (-1.5%) CPS 17 636 17 557 (-0.4%) The worst case improves more than the average: no core dropped below 76% with rebalancing, against a low of 47% without it. At high connection counts the hash is already even, so the balancer only costs its own bookkeeping. Off by default (`connection_rebalance`) since it trades a little at the high end for a lot at the low end, and which matters depends on the deployment.
…fer-pool wiring `Balancer::slot_for_self` became dead once handoffs carried their slot with them, and `PooledBufs::pair` once the relay borrowed the buffer fields directly.
…s reactor
`TcpListener::poll_accept` hands back a `TcpStream` already registered with the
accepting core's reactor. Moving that object to another core left the
registration behind, so every read and write on a rebalanced connection went
through the accepting core's epoll and woke the owning core across the channel
-- for the whole life of the connection, not just at handover.
That turned the rebalance into a net loss whenever the hash was already even.
Measured on passthrough small-request at 50 connections, where only ~50 accepts
happen and migrations should be invisible:
rebalance off 295 444 rps
rebalance on, code path but no 295 033 rps (+0.2%)
migrations (slack raised to 999)
rebalance on, migrating 288 825 rps (-1.9%)
The middle arm is what isolates it: every line of the balancer runs, and it
costs nothing. The cost was per *migrated connection*, and scaled with how
aggressively the threshold migrated -- which is what a permanently misplaced
reactor registration looks like, not what a channel send looks like.
Handing over the plain `std::net::TcpStream` deregisters on the way out and
re-registers on the way in. The penalty disappears: +0.4% at 50 connections
(was -2.5%), +0.8% on TLS terminate (was -2.1%), and the gain at 16 connections
grows from +4.0% to +10.7%.
…ng a fixed slack
A fixed slack of 2 is the right amount of hysteresis at 4 connections per core
and far too little at 12: it keeps migrating on differences that are noise at
higher load. Comparing thresholds on passthrough small-request:
connections 16 50
slack 2 236 646 290 542
slack 1 240 190 285 047
slack 0 239 335 284 847
proportional 246 216 287 714
and the worst-loaded core at 16 connections goes 81% (slack 2) -> 95%
(proportional). `least + 1 + least/4` migrates on relative imbalance, so it
still reacts when one core has 6 connections and another has 1, and stops
reacting when they have 13 and 12.
Also tried and dropped: capping accepts per scheduler turn the way HAProxy's
`maxaccept` does, and pushing every connection through the handoff channel even
when it stays local (HAProxy does this deliberately, for cache locality). Both
measured within noise here once the threshold was right.
This was opt-in because handing a connection over cost 2-3% wherever the `SO_REUSEPORT` hash was already even. That cost was a bug, not a trade -- the migrated socket kept its readiness registration on the accepting core's reactor -- and with it fixed there is no longer a workload where the rebalance loses. Measured on a 4-core gateway, 3-5 runs per arm after warmup, with bulk throughput re-run interleaved (off/on/off/on, n=5 each) because a single pass had suggested a 2% regression that turned out to be noise: | workload | off | on | | |---------------------------------------|-----------:|-----------:|-------:| | passthrough small-request, 8 conns | 113 350 | 143 341 | +26.5% | | passthrough small-request, 16 conns | 231 043 | 258 787 | +12.0% | | passthrough small-request, 50 conns | 294 937 | 295 311 | +0.1% | | TLS terminate small-request, 50 conns | 244 304 | 246 228 | +0.8% | | TLS terminate, connections/s | 41 024 | 43 019 | +4.9% | | passthrough, connections/s | 17 431 | 17 319 | -0.6% | | passthrough, bulk throughput | 12.10 GB/s | 12.16 GB/s | +0.5% | The gain concentrates where the hash has fewest connections to spread: at 16 connections the quietest core goes from 63% to 97% busy. At 8 and 16 connections this also moves us from behind HAProxy to level with it (143 341 vs 140 278, and 258 787 vs 261 407).
The byte gate was a proxy for "is this connection worth the per-connection setup cost", and it works for request/response traffic where byte volume and message size move together. LLM token streaming is where the proxy breaks: 4000 tokens of ~64 B is ~256 KB total, so the connection is classified bulk, while its actual shape -- many tiny writes -- is exactly what the gate exists to exclude. Worse, the classification arrives late: 64 KiB at 40 tok/s is ~25 s, so every stream spends its first quarter on the copy path and short conversations never promote at all, making the path taken depend on prompt length. Add an independent duration gate. The two are complementary rather than alternatives: bytes catches high-rate connections within milliseconds but is blind to long-lived low-rate streams; duration catches those but is blind to short high-rate ones that finish before it fires. Whichever fires first engages. Check the gate only at the tail of the relay loop, where both directions have completed their write_all and nothing is buffered in userspace -- the same point the byte check already used, so the handover boundary is unchanged. A timer arm inside the select! would also be safe but would promote merely-idle connections, allocating a pipe for a stream with nothing to move. Replace the four flat fields with two optional sections. `0` previously had to mean both "engage immediately" and, once a second gate exists, "this gate is disabled"; with an Option per gate and the whole section optional, every state has exactly one representation: absent section = disabled, empty section = engage immediately, either gate set = engage when it fires. This also folds away the separate `*_enabled` booleans, which would otherwise be a second spelling of "never engage". None of these fields are upstream yet, so there is no compatibility cost. Also add serde_duration::option for Option<Duration> config fields, with tests.
splice needs a pipe because the kernel has no socket-to-socket path: the zero-copy is page-reference passing and the pipe is the container those references live in. A bidirectional relay therefore holds one pipe per direction, and since a pipe is unidirectional and stateful the two cannot be shared, so a spliced connection pins four descriptors. Held for the connection's lifetime that is proportional to connections rather than to work: LLM token streaming moves a ~64 B record every 25 ms and spends over 99% of the connection idle, yet keeps all four. Measured on the streaming bench, 50k connections with 10k active streams reached 140 031 descriptors, and every connection eventually crosses the engage gate, so the steady state for 50k streams is 300k descriptors. Park the pipe while the source is dry. The release point is provably safe: the relay only waits for readability once the previous chunk has been fully drained into the destination and the fill attempt that just returned WouldBlock added nothing, so the pipe is empty exactly where it is handed back. The saving is sharing, not elimination -- parked pipes stay open in the pool. The bound moves from 4 * connections to 2 * PIPE_POOL_MAX * workers plus whatever is genuinely in flight, so a 32-worker gateway tops out near 4k descriptors instead of 300k. Pipes are only created when the pool is empty, so the live count converges on peak concurrent in-flight chunks rather than churning pipe2/close. Behind release_idle_pipes so the two behaviours can be benchmarked against each other. Tests cover it in descriptors rather than through the pool as a proxy, counting this thread's live pipes exactly (process-wide /proc/self/fd races with tests on other threads). Eight idle connections cost 2 descriptors with release on and 32 with it off. Also covered: payload integrity across repeated park/reacquire cycles, which is where a stale or half-drained pipe would corrupt the stream.
Both knobs now carry the numbers they were introduced on, in the same shape as the other measured config docs in this file. The descriptor result is the exact one: 2000 spliced streaming connections cost 8000 pipe descriptors with release off and 8 with it on, reproduced on every run. Latency is recorded as the null result it is -- the fastest single run of seven was a release-off run, so the only defensible claim is steadier tails, not throughput.
…on ones The earlier numbers were taken at 2000 connections because the bench client was being invoked without RAMP, which leaves its ramp batch size at a small default and dribbles out ~40 connections/s. With RAMP/RAMP_MS set as a pair the client reaches 50k in 6.6 s, so the descriptor claim can now be checked at the scale it was made for. It holds, and it is flat: 8 pipe descriptors for the whole gateway at 2k connections and at 50k alike, against 53 864 with release off. Latency is recorded as the wash it is rather than the win the first single run suggested. kTLS is measured for the first time here. It does nothing for latency on 64 B records, which was the prediction, but it cuts RSS 349 -> 206 MB because a spliced kTLS connection never brings the payload into userspace and so needs no relay buffer. That was not predicted and is the only real argument for it on this traffic.
The one failure mode this change could plausibly have introduced was never measured: a connection that is never idle never reaches the release point, so bulk traffic could hold every pooled pipe and force each token to pay a pipe2/close pair. Measured with both classes on the same gateway, 10k token streams alongside 256 bulk connections saturating passthrough at 12.8 GB/s. The pool holds at 8 descriptors against 9024 with release off, bulk throughput is identical, and both classes come out marginally faster with release on. The reason it does not degenerate is that even a flat-out bulk connection idles ~62 us between chunks and releases in that window. Getting the saturated case took two attempts: at 96 bulk connections the gateway was only asked for 6.1 GB/s against a ~12 GB/s ceiling, so nothing was backlogged and the test proved less than it appeared to.
A kernel built without CONFIG_TLS accepts the `[core.proxy.ktls]` section and then rejects every offload, and the two modes fail differently. Immediate offload fails at the handshake, so the gateway serves nothing -- bad, but loud. The gated offload is worse: the connection is served from userspace up to the threshold and only fails at the gate, so the client gets HTTP 200 with the body truncated at exactly the gate size, and the gateway logs it as a per-connection error rather than a misconfiguration. Measured on a host with the tls module blocked, `after_bytes = 65536`: before GET /100m -> 200, 65536 of 104857600 bytes, silent truncation after GET /100m -> 200, 104857600 bytes, one warning at startup So probe once at startup instead: establish a throwaway loopback connection and set TCP_ULP on it (the ULP is only accepted on an established socket, so probing a fresh one would report ENOTCONN either way). If the kernel refuses, clear the section and warn. This runs before the acceptor is built, which is also what decides whether rustls extracts session secrets at all -- so the fallback does not leave secret extraction enabled for an offload that can never happen. This is not hypothetical for dstack: the guest OS image ships no tls.ko and no CONFIG_TLS, so a gateway running inside a CVM hits exactly this path today. Verified on Linux 6.8 with the tls module present (kTLS still engages, TlsTxSw +1 for a 100 MiB transfer, no warning) and blocked (both modes fall back to userspace rustls, full body delivered, zero per-connection offload errors).
Neither acceleration option is simply on or off, and nothing surfaced that. kTLS can be cleared at startup by the capability probe, and both kTLS and splice engage per connection only once their gate fires, so the config file does not say what is running -- and the difference is worth 2-4x in per-connection memory. The only signals available were one startup warning and the host-global /proc/net/tls_stat, which is not per-gateway and has no splice equivalent. Adds ProxyAccelStatus to the Status RPC and a Data Path panel to the dashboard: the effective mode of each option, plus three counters -- connections offloaded to the kernel, offloads that failed, and connections that entered a splice relay. Failed offloads are separate from "not accelerated" on purpose: non-zero there means connections are being dropped or truncated, not just running slower. Counters are relaxed atomics bumped once per connection, on the same order as the existing NUM_CONNECTIONS. Splice is counted in splice_bidirectional, the one funnel both the passthrough gate and the post-kTLS handover pass through. Verified against traffic with a known shape (2 small terminate requests under the gate, 1 large one over it, 1 large passthrough), across every state: ktls=off ktls=off splice=after 64 KiB or 5s 0/0/1 ktls=adaptive ktls=after 64 KiB or 5s splice=after 64 KiB or 5s 1/0/2 ktls=immediate ktls=immediate splice=after 64 KiB or 5s 3/0/4 splice=off ktls=off splice=off 0/0/0 no TLS ULP ktls=disabled (...) splice=after 64 KiB or 5s 0/0/1 (offloaded/failed/spliced). The adaptive arm offloads only the connection that crossed the gate; the immediate arm offloads all three; splice counts one extra for the passthrough connection in every arm that has splice on.
The gateway CVM app image is built for x86_64-unknown-linux-musl, and ktls 6.0.2 does not compile for it: error[E0063]: missing field `__pad1` in initializer of `cmsghdr` error: cannot construct `msghdr` with struct literal syntax due to private fields The crate builds both structs with struct literal syntax. That is fine on glibc, where they have exactly the members it names, but musl declares msg_iovlen and msg_controllen as int/socklen_t followed by explicit padding, and libc models that with private __pad1/__pad2 members a literal cannot name. So this is a permanent property of musl's ABI rather than a libc regression, and 6.0.2 is the latest published release -- there is no version to bump to. rustls/ktls#70 fixes it by building both from std::mem::zeroed() field by field. It has been open since 2026-05-04. Rather than point the dependency at the contributor's fork -- a moving, non-crates.io source for the crate that installs session keys into the kernel, in a build that otherwise pins everything down to package versions and image digests -- the 6.0.2 release is vendored with only that patch applied. Each hunk is marked, dev-dependencies are dropped, and vendor/README.md records the provenance and the condition for deleting it again. ktls was the only thing blocking the musl build (confirmed with --keep-going). Verified by running both binaries against the same backend: a 100 MiB transfer through the terminate path checksums correctly, kTLS actually engages (ktls_offloaded=2, ktls_offload_failed=0) and TlsDecryptError stays at 0 for the glibc and the static musl build alike. The patched code is on the close_notify path, so compiling was not the interesting half.
rust-checks runs clippy with `-D clippy::expect_used -D clippy::unwrap_used`,
which the perf work tripped in four places, plus one field left dead by the
timer removal.
* PooledPipe::rd/wr: the ends are Option only so Drop can move them into the
pool, so both are always present for a borrow. Uses or_panic, the helper
this crate already uses for the same situation, and says why in a comment.
* PooledBufs::get: min().max() is a clamp; write it as one.
* connect_multiple_hosts: the single-candidate fast path checked len() == 1
and then unwrapped next(). Take the first candidate up front and branch on
whether any remain, which drops the unwrap and the second length check;
the racing path now iterates once(first).chain(rest).
* Timeouts::write: dead since the per-operation timers were replaced by the
connection-level progress watchdog, which catches a stalled write through
`idle` instead. The field is kept so existing configs -- and the CVM
entrypoint's TIMEOUT_WRITE -- still parse, with a doc comment saying it no
longer does anything. Whether that knob should be retired is a separate
call from making CI green.
The vendored ktls is a path dependency, so cargo does not cap its lints the way
it would for a crates.io one; it carries a crate-level allow for the same two
lints, since it is vendored to be patched minimally rather than rewritten.
`cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used
--allow unused_variables` is now clean across the workspace, `cargo fmt --check`
passes, and the 43 gateway tests pass. Re-ran the acceleration verification
after the change -- the passthrough fast path is on it -- and every arm reports
identical counters to before.
CI runs `cargo test --all-features`, which applies to workspace *members*. Making vendor/ktls a member therefore enabled its `ring` feature alongside `aws_lc_rs`, which the crate declares mutually exclusive: error[E0252]: the name `cipher_suite` is defined multiple times error: The 'ring' and 'aws_lc_rs' features are mutually exclusive As a crates.io dependency its features came from the dependency declaration and nothing could turn `ring` on, so vendoring changed feature resolution rather than anything about the crate. Excluding it restores that: it stays a path dependency and still builds, but `--all-features` no longer reaches it. Dropping the `ring` feature from the vendored manifest would also compile, but the gateway exposes `tls_crypto_provider = "ring"` and ktls matches rustls' cipher suites against whichever provider it was built with, so that would quietly break kTLS for ring deployments. Verified with the commands CI actually runs, not the narrower ones I used the first time: `cargo check --all-features`, `cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables`, `cargo fmt --check --all`, `./run-tests.sh`, and the musl release build.
…e reply
Both gated relays treated one direction reaching EOF as the end of the
connection, and the splice one also shut down the wrong half:
if n == 0 { finish_one!(br, aw, bufs.b); }
// macro: aw.shutdown(); then loop { br.read() -> aw.write_all() }
`ar` reaching EOF means the *client* stopped sending. The EOF owes to the app,
via `bw`. Shutting down `aw` instead tells the client the response is over and
then writes the app's reply into the half just closed, so a client that ends its
request with `shutdown(SHUT_WR)` gets a successful, empty response. Line 347 had
the mirror image of it, losing a request instead of a reply.
`adaptive_ktls::relay_until` had the same shape without the swap: the first EOF
from either side broke straight to `Phase::Eof` and the caller dropped both
sockets, so anything still in flight the other way went with them.
Both now propagate the EOF to the peer of the direction that ended, drain the
opposite direction, and only then close. Nothing changes once a connection is
spliced or offloaded -- `splice_one` already half-closes correctly -- so this is
confined to the pre-gate phase.
Four tests, each of which fails without the change:
response_survives_a_client_half_close_before_the_gate (splice, ktls)
request_survives_a_backend_half_close_before_the_gate (splice)
request_survives_an_app_half_close_before_the_gate (ktls)
`relay_until` being generic over the client stream is what lets the kTLS ones
run a plain socket in place of the TLS one and exercise the same logic without a
handshake.
Reachable whenever a gate is configured, i.e. exactly the configuration
gateway.toml recommends (`after_bytes = 65536`, `after_duration = "5s"`). The
existing splice tests covered pipe accounting and EOF propagation on a fully
closed connection, which is why this got through.
…recovered
Two problems in the rebalancer, both about what happens when a core stops
behaving.
The handoff channel was unbounded. A core only receives connections while it is
the least loaded, so in steady state it never backs up -- but if one stops
draining, every other core keeps succeeding at `send`, and each handed-over
connection then sits unserved until `timeouts.total` (5h by default) while its
`CoreSlot` keeps that core looking busier than it is. The queue and the memory
behind it only grow. It is now bounded at 1024 with `try_send`, so a full queue
means "this core is not taking work" and the sender keeps the connection.
Two failure paths also had comments claiming the opposite of the code:
// Cannot deregister: keep it here rather than lose the connection.
Err(_) => return None,
`TcpStream::into_std` takes `self`, so on error the socket is already closed and
`None` tells the caller it was handed away -- the connection is lost, not kept.
`TcpStream::from_std(raw).ok()` did the same under "keep the connection rather
than drop it". Neither is recoverable, so both now log what actually happened
instead of describing a fallback that was never there. The paths that *are*
recoverable -- a full or closed queue -- really do keep the connection now.
Also: the slot for the target core is claimed after `into_std` succeeds rather
than before, so the failure path no longer briefly accounts a connection to a
core that will never see it, and `start_thread_per_core`'s comment no longer
justifies ordered binding by the CBPF steering program that `reuseport`'s own
docs say was abandoned.
…LS handover
`buffer_size` is the one field this PR changes a default on (8 KiB -> 64 KiB) and
was the only `ProxyConfig` field with no doc comment at all. It now records what
the 8x costs: `2 * buffer_size` of address space per userspace-relayed
connection, measured at ~52 KB RSS per connection at 2 000 concurrent streaming
connections, against ~12 KB on the kTLS path where the payload never enters the
process.
The kTLS handover flushed two different kinds of data through one loop:
for chunk in [drained.unwrap_or_default(), buffered] { ... }
`drained` is plaintext rustls decrypted past the handshake and owes to the app.
`buffered` is whatever raw *ciphertext* was left in the SNI sniff buffer --
forwarding it would hand the app TLS records to interpret as application data.
A completed handshake always consumes the sniff buffer, so this is unreachable
rather than merely unlikely, which is exactly why it should fail loudly instead
of being discovered as a corrupted stream.
`s.split_at(s.len() - 1)` is a byte index, so a multi-byte unit lands inside a
character and panics in `core::str`: `parse("5分")` aborted the config load
rather than reporting a typo. Splits on the last character now.
The unit multiply was also unchecked, so `18446744073709551615d` wrapped to a
short duration in release builds -- a timeout far tighter than the operator
asked for, which is the worst way for this to fail. Checked, with an error.
Both are pre-existing, but this PR restructured the function and added five
tests around it without covering either.
The gated fast paths bypass `io_bridge`, and `io_bridge` was where both the idle watchdog and the TLS-aware close lived, so enabling splice or kTLS silently dropped two behaviours the config still claims to control. **Idle timeout.** Measured with `idle = 3s`, a 1 MiB response read to completion, then 15s of silence: terminate, buffered reaped terminate + kTLS + splice ALIVE passthrough, buffered reaped passthrough + splice ALIVE And it did not need the gate to fire: with the gate at 10 MiB and a 1 KiB response -- so the connection never left the pre-splice phase -- both gated paths still stayed alive, because `splice_bidirectional_after` and `relay_with_adaptive_offload` route around `io_bridge` from the first byte. Merely *configuring* a section disabled the idle timeout for that whole path. The only remaining bound was `timeouts.total`, five hours by default -- on exactly the long-lived streaming connections the gates exist to capture. The watchdog is now a shared `proxy::idle::IdleWatchdog` that all three relay shapes use: the buffered bridge and the two pre-gate relays poll it as one more `select!` arm, and a spliced connection -- which has no loop of its own, since its bytes never enter this process -- races it against the transfer. `splice_one` bumps a relaxed counter once per chunk, not per byte, which is nothing next to the splice syscall it accompanies. **close_notify.** `splice_one` ends a direction with a bare `shutdown(fd, SHUT_WR)`. On a kTLS socket that is a FIN with no TLS close, which is indistinguishable from a truncation attack to a client that checks: userspace rustls client saw: clean_eof kTLS + splice client saw: TRUNCATED (no close_notify) `KtlsStream::poll_shutdown` would have sent it, but the offload path takes the raw descriptor and never goes through that type again. The destination's close is now described by a `CloseKind`, and the kTLS side sends `close_notify` first. That needed a re-export in the vendored crate, which upstream keeps private because only `KtlsStream` uses it -- marked as a dstack patch like the other one. Verified: every arm above now reaps (`bridge error: idle timeout` in the log) and reports `clean_eof`, byte-for-byte identical payloads. `data_timeout_enabled = false` still opts out, and the accel counters and 100 MiB checksums are unchanged on both the glibc and musl builds.
`connection_rebalance` is only read inside `start_thread_per_core`, so with `thread_per_core = false` -- or after the `SO_REUSEPORT` probe falls back, where the existing warning mentions only `thread_per_core` -- it is silently ignored. The config docs say "Requires `thread_per_core`" and nothing enforced it, so an operator who set one but not the other got no signal either way. Says so now, on the shared-runtime path. Not an error: a shared work-stealing runtime genuinely has nothing to rebalance, so the right outcome is to serve traffic and say the knob is inert.
The gateway had substantial tests -- `test_suite.sh` is 65 KB -- and no
workflow referenced any of them. Nothing in CI ran a gateway process at all,
which is how a response-dropping half-close, a silently disabled idle timeout
and a missing close_notify all reached review.
`test_proxy.sh` starts a real gateway and asserts on what reaches the wire,
across every combination of the two gated optimisations and both proxy paths:
data path payloads survive byte-for-byte, under and over each gate,
single and concurrent
close the app closing arrives as an orderly TLS shutdown, kTLS
offload included
timeouts.idle the watchdog reaps on buffered, spliced and kTLS relays,
and `data_timeout_enabled = false` still opts out
kTLS engagement offload happens, only once the gate fires, no decrypt errors
fallbacks a kernel without the TLS ULP warns and serves untruncated;
an inert connection_rebalance warns
runtime modes every thread_per_core / connection_rebalance combination
Status RPC the accel counters reflect the traffic that ran
48 assertions locally. Every capability it cannot get -- no TLS ULP, no
privileges for `wg show` -- is reported as SKIP with the reason rather than
quietly passing, since a test suite that reports success for work it did not do
is worse than one that is missing.
Three things the design had to get right, each learned by getting it wrong:
* The gateway only needs *a link* with its configured wg interface name to
start, not a working WireGuard device, so the suite falls back to a dummy
link and runs the whole data path without the wireguard module. Only the
Status RPC needs the real thing, and that group probes the RPC rather than
guessing from the link kind -- it also needs root for `wg show`.
* Restarting the gateway ~25 times races its own listeners, so teardown waits
for the ports to come back. A preflight check refuses to start when they are
already busy: a stale listener does not fail cleanly, it answers the probes
with the wrong config and scatters unrelated assertion failures.
* Half-close is *not* covered here. Every client of this gateway speaks TLS,
and a TCP half-close mid-TLS is a truncation rather than an orderly one --
confirmed by pointing the same probe at the origin with no gateway in the
path, where it fails identically. It stays a unit test, and the script says
so.
The workflow is path-filtered to the gateway and the vendored ktls, records the
kernel's TLS and AES-GCM capabilities before running so a skipped group is
visible in the log, and uploads the work dir on failure.
dstack-gateway can offload TLS record encryption to the kernel, but no dstack
guest image could accept it: the 0.5.11 image ships 67 modules and `tls.ko` is
not one of them, and no fragment in either backend set `CONFIG_TLS`. A gateway
running inside a CVM therefore always fell back to userspace rustls.
Adds it to both kernel configs, plus the crypto the record layer needs:
CONFIG_TLS the `tcp_ulp = "tls"` upper-layer protocol
CONFIG_CRYPTO_GCM the AES suites the gateway negotiates
CONFIG_CRYPTO_AES ditto
CONFIG_CRYPTO_AES_NI_INTEL accelerated GCM (already set for mkosi)
CONFIG_CRYPTO_CHACHA20POLY1305 so a client that picks it fails at
configuration time, not per connection
Built in rather than a module. The two backends package modules differently --
mkosi runs `modules_install` and ships everything built, while the yocto image
lists them one by one (`kernel-module-tun`, `kernel-module-fuse`, ...) -- so
`=m` would need a matching `kernel-module-tls` entry in
`dstack-rootfs-base.inc`, and forgetting it fails at runtime rather than at
build time. That is the failure this commit exists to remove, so it is not a
good thing to reintroduce one level down.
`CONFIG_TLS_DEVICE` is deliberately left off: it hands session keys to the NIC,
which a confidential guest has no reason to trust and no CVM NIC offers.
This does not turn anything on. kTLS stays opt-in in `[core.proxy.ktls]`, and
the gateway still probes for the ULP at startup and falls back with a warning
if it is absent -- which is what keeps this safe for images built before this
change.
AES-NI matters more than it looks: measured on Emerald Rapids across three
kernels on identical hardware, kTLS bulk throughput is -18.3% on 6.1 (only
`generic-gcm-aesni` registered), -2.1% on 6.12 and +20.3% on 6.17 (both
register `generic-gcm-vaes-avx10_512`). The guest kernel is 6.18.
mkosi's `check-kernel-config.sh` asserts every symbol in the fragment against
the generated `.config`, so all five are now build-time gated there.
kvinwang
force-pushed
the
gateway-perf-clean
branch
from
July 27, 2026 06:49
262751b to
3a9b6f3
Compare
…sponse `splice()` on a kTLS socket refuses any record that is not application data and reports it as EINVAL. That is how a client's `close_notify` arrives once the socket belongs to the kernel, and `splice_one` treated it as fatal: connection error: ktls splice error: splice src->pipe failed: Invalid argument `try_join!` then cancelled the *other* direction, so a client ending its request threw away the response the app had already written. Same shape as the pre-gate half-close bug, one layer down and only once kTLS has actually engaged, which is why the earlier fix did not cover it. A non-data record is the end of the data stream for a byte relay, not a failure, so it now breaks the loop like any other EOF and the opposite direction drains. `CloseKind` describes both endpoints rather than just the destination, since whether EINVAL means "control record" depends on which side is kTLS. Found by `proxy/tlsclient.py`, added here: `ssl.SSLSocket` cannot half-close a TLS connection at all -- `unwrap()` waits for the peer's `close_notify`, which the peer will not send until it has finished replying, and `shutdown(SHUT_WR)` sends a bare FIN that mid-TLS is a truncation the peer is right to reject. Driving the state machine over memory BIOs does it properly: `unwrap()` writes the alert into the outgoing BIO before it raises, so flushing that and declining to finish the handshake is exactly a half-close. The half-close group is back in the suite, 5 arms x 2 paths. Before this fix `ktls-immediate / terminate` returned 0 bytes and everything else passed, which is what pointed at the offload path. Verified the fix does not truncate legitimate traffic: 1 KiB, 1 MiB and 8 MiB through kTLS+splice all arrive complete with matching digests and zero splice errors. Suite is 58 passed, 0 failed, 2 skipped; 49 unit tests pass.
`take_sni` reads raw wire bytes off the socket before the proxy knows where the
connection is going, so those bytes live in a `Vec` that splice cannot see. Each
path has to account for them, and one did not.
Passthrough is fine: it replays the buffer to the app with `write_all` before
splice takes the sockets. The terminate paths feed it to rustls through
`MergedStream`, which yields it ahead of the socket and empties itself once
consumed -- so at the kTLS handover the remainder is normally empty.
"Normally" was doing real work there. The immediate offload path checks it; the
adaptive one reached the socket through
impl From<MergedStream> for TcpStream {
fn from(s: MergedStream) -> Self { s.into_parts().1 }
}
which drops the remainder on the floor, with a doc comment asserting the
invariant instead of enforcing it -- and an `IO: Into<TcpStream>` bound that
made it look unremarkable at the call site.
That conversion is gone. Both paths now go through a `SocketParts` trait that
hands back `(remainder, socket)`, and both refuse a non-empty remainder. There
is no safe alternative to refusing: the bytes are ciphertext, so they cannot go
to an app expecting plaintext, and once the socket belongs to the kernel they
cannot go back either. Forwarding them would hand the app TLS records to parse
as application data, which is a much worse thing to debug than a failed
connection.
Still unreachable in practice -- rustls consumes the whole ClientHello during
the handshake, and the shipped TLS 1.2 config has no early data for a client to
get ahead with. The point is that it is now unreachable *by construction*
rather than by convention.
Two tests pin it: a non-empty remainder is surfaced, and a consumed buffer
leaves none. 51 unit tests and 58 integration assertions pass.
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Reported by Codex review, confirmed and reproduced.
Every relay leaves its main loop when one direction hits EOF and drains the
other in a plain loop with no timeout on the reads. `timeouts.idle` therefore
stopped applying at exactly the point a connection becomes cheapest to abuse:
a client sends a request, half-closes, and a backend that accepts it and stays
silent holds the connection until `timeouts.total` -- five hours by default,
against a ten minute idle -- with its relay buffers still allocated.
This is a regression from this PR, not pre-existing. Before the watchdog
replaced the per-operation timers, `step()` read through
`timeout(self.cfg.timeouts.idle, ...)`, which covered the drain too. Moving
coverage into the `select!` loop left the phase that runs after it bare.
Three sites, all of them:
io_bridge.rs the `Rest::A2b` / `Rest::B2a` drain after the select loop
splice.rs `finish_one!` in the pre-gate relay
adaptive_ktls.rs `finish_one!` in the pre-gate relay
The drain now polls the same watchdog. `io_bridge` freezes the finished
direction's progress and keeps adding it, since the watchdog samples a
monotonic counter and switching to the survivor alone would look like the
counter went backwards.
`splice_one` needed nothing: `splice_bidirectional` races the whole `try_join!`
against `wait_until_stalled`, so one direction ending leaves the join pending
and the watchdog still fires.
Measured with `idle = 3s`, a half-closed request against a backend that never
replies:
before after
buffered open at 25s reaped at 3.8s
splice engaged open at 25s reaped at 3.7s
splice gated open at 25s reaped at 3.7s
ktls gated open at 25s reaped at 3.8s
Four integration assertions cover it, against a new `/blackhole` origin
endpoint that drains the request and then holds the connection open. The suite
is 62 passed, 0 failed, 2 skipped; 51 unit tests pass.
Second round of the same Codex finding, and the first fix only covered half of
it. The drain loop put `$r.read()` in the `select!` with the watchdog but left
$w.write_all(&$buf[..n]).await?;
outside it. A backend that floods while the client stops reading fills the
send buffer, `write_all` blocks, and the watchdog goes unpolled again -- the
mirror image of the silent-backend stall and just as good for holding a
connection to `timeouts.total`.
`write_all` cannot simply move into the `select!`: it is not cancel-safe, so
losing the race would drop an unknown number of already-written bytes. Single
`write` calls are cancel-safe -- tokio guarantees nothing was written when
another branch wins -- so the partial-write loop is driven here instead, with
the watchdog racing each call. `moved` advances per partial write, so a peer
that drains slowly still counts as making progress and is not reaped for being
slow.
`io_bridge` needed nothing, for the reason Codex gave: `OneDirection::step()`
is a cancel-safe state machine that performs one operation per call, so racing
`step()` already covered its writes. That difference is what the measurement
showed.
Measured with `idle = 3s`, a half-closed request against a backend flooding
64 MiB while the client reads nothing:
before after
buffered reaped, 4.1 MB drained unchanged
splice engaged open, 67 MB drained reaped, 4.1 MB
splice gated open, 67 MB drained reaped, 4.2 MB
ktls gated open, 67 MB drained reaped, 4.1 MB
The read-stall case still reaps at 3.8s and an 8 MiB transfer is still intact,
so this does not reap connections that are merely slow.
Four more assertions against a new `/flood/<n>` origin endpoint, one per arm.
Suite is 66 passed, 0 failed, 2 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance work on
dstack-gateway's reverse proxy path, developed and benchmarked in session a31279f8, with a follow-up round of measurement and review in session 6de7fae5.Rebased onto
masterwith only the gateway proxy commits (no test-plan / full-plan history).Gateway proxy changes
splice(2)zero-copy relay for TLS passthrough (Linux), gated on bytes and/or elapsed timeJoinSetwhen only one upstream candidateProxyAccelStatuson theStatusRPC and a Data Path panel on the dashboardConfig (
[core.proxy])tcp_spliceandktlsare optional sections, not boolean flags. Omitting the section disables the optimisation; an empty section engages it from the first byte;after_bytesandafter_durationare independent gates and whichever fires first engages it.buffer_size655368192; costs2 × buffer_sizeper userspace-relayed connectionthread_per_coretrueconnection_rebalancetruethread_per_core[core.proxy.tcp_splice]after_bytes,after_duration,release_idle_pipes[core.proxy.ktls]after_bytes,after_duration; gated mode also requirestcp_spliceBehaviour changes worth reviewing
idletimeout killed a connection when one direction went quiet; the watchdog now requires both to stall. More correct for a proxy, but it changes when connections are reaped.timeouts.writeis inert. Superseded by the watchdog; the key still parses (the CVM entrypoint sets it fromTIMEOUT_WRITE) but has no effect. Retiring it is a separate call.thread_per_coreandconnection_rebalanceship enabled.Benchmark notes
Validated on dedicated GCP machines against HAProxy where applicable; both TLS passthrough and terminate paths exercised.
epoll_ctlcost is ~0 in keep-alive windows; ~0.04–0.15% of gateway CPU in the CPS worst case.release_idle_pipes.kTLS: measured trade-off, opt-in on purpose
Re-measured locally on a Xeon Gold 6530 (kernel 6.8), gateway pinned to 4 cores, 3 interleaved rounds:
The connection-setup cost reproduces the numbers in the commit messages. The
bulk figure did not, so it was re-measured on three GCP VMs of identical shape
(c4-standard-16, Xeon Platinum 8581C, Emerald Rapids -- same microarchitecture
as the local host), varying only the kernel:
gcm(aes)drivergeneric-gcm-aesni(prio 400)generic-gcm-vaes-avx10_512(prio 800)generic-gcm-vaes-avx10_512(prio 800)(MB/s; MB per gateway CPU-second tracks it to within 0.5 points in every row.
3 interleaved rounds, gateway pinned to 4 cores, same static musl binary on all
three.)
So the direction was right and the detail was not. The regression is real on a
pre-VAES kernel -- 6.1 reproduces the -24% measured locally on 6.8 -- and the
VAES driver does remove it. But 6.12 only reaches parity: the +20% needs 6.17,
so something past the driver table matters too, and "6.11 fixes it" was too
simple a claim to have made from the driver tables alone.
For dstack this lands well: the guest OS builds 6.18, past every row here.
Two things worth recording from the measurement itself.
perfon the local6.8 host puts crypto at 13.8% of gateway cycles with rustls
(
aes_gcm_encrypt_avx512) against 36.7% with kTLS (key_256_enc_update4),which is the mechanism. And the first run of this experiment measured a
meaningless +-0.5% on both Debian VMs because those images ship no
tlsmodule: the startup probe correctly disabled kTLS and the arms were rustls
against itself -- an unplanned end-to-end check that the probe degrades
cleanly rather than truncating.
The substantive, portable win is memory, not throughput. Off by default on all counts.
Vendored
ktlsktls6.0.2 does not compile forx86_64-unknown-linux-musl(the CVM app image target): it buildslibc::cmsghdr/msghdrwith struct literals, and musl's ABI has private padding members. It is the latest release, sodstack/vendor/ktls/carries 6.0.2 plus rustls/ktls#70 and nothing else.dstack/vendor/README.mdrecords the provenance and the condition for deleting it.Guest kernel
CONFIG_TLSwas absent from both OS backends, and the 0.5.11 image ships 67modules without
tls.ko-- so a gateway inside a CVM could never accept theoffload and always fell back. This PR enables it (plus
CONFIG_CRYPTO_GCM,AES,AES_NI_INTEL,CHACHA20POLY1305) inos/yocto/.../dstack.cfgandos/mkosi/components/kernel/kernel.config, built in rather than as a modulebecause the two backends package modules differently.
CONFIG_TLS_DEVICEstaysoff: it would hand session keys to the NIC.
Nothing is turned on by this. kTLS remains opt-in, and the startup probe still
falls back with a warning on images built before the change.
Test plan
cargo check --all-features,cargo clippywith the CI lint set,cargo fmt --check --all,./run-tests.shthread_per_core+ rebalance)tcp_splice/ktlscombination, verified against traffic of a known shape via the new accel counterstlsmodule blockedthread_per_core,connection_rebalance) for production risktimeouts.write/TIMEOUT_WRITEshould be retired