[liveinvalley]

Issue

Fixes #685 (suggestion 1).

Problem

Since Go 1.23, net/http no longer cancels an in-flight dial when the request is canceled (commit 334ce510), so http.Client.Timeout does not bound the dial. makeRestcall (used by CreateControlChannel and CreateDataChannel) ran on a clone of http.DefaultTransport, whose net.Dialer{Timeout: 30s} is split evenly across the resolved addresses. With two resolved ssmmessages addresses and the first one black-holed, the dial spends 15 s on the first address, which is the whole 15 s client budget. The request never reaches the second address, and the control channel cannot be re-established however often the agent retries. See #685 for the packet capture.

Change

In agent/session/service/service.go, a new newMgsHttpClient builds the client for the MGS REST calls:

• Transport.DialContext uses a dedicated net.Dialer with Mgs.ControlChannelDialTimeoutMillis (default 20 s, so 10 s per address with two addresses). This is the same setting the control-channel websocket dial has used since 3.3.5390.0, so the token request now fails over the same way the websocket dial does.
• Transport.ResponseHeaderTimeout = 15s bounds the wait for the service response separately.
• Client.Timeout becomes dial timeout + TLS handshake timeout + 15 s (45 s by default), as a backstop.
• ControlChannelDialTimeoutMillis = 0 (the existing opt-out) keeps the previous behavior: the default transport and a 15 s client timeout.

Trade-off: a single REST attempt can now take up to 45 s by default instead of 15 s, but only when connecting is slow (an unreachable first address or a stalled TLS handshake). A service that accepts the connection and does not answer still fails after 15 s, as before, through ResponseHeaderTimeout.

Testing

• New unit tests in agent/session/service/service_test.go check the client configuration for the default, custom and disabled (0) dial timeouts, an httptest round trip, and that ResponseHeaderTimeout is enforced.
• go test ./agent/session/service/... ./agent/session/controlchannel/... and go vet ./agent/session/service/ pass. make checkstyle reports no new findings (the only go vet finding, in agent/session/crypto/blockcipher_test.go, exists on mainline).
• Black-hole scenario from CreateControlChannel request never reaches the second ssmmessages address when the first is black-holed #685 against ssm-agent-worker.exe built from c6c4ce7 (same setup as run fault2 in CreateControlChannel request never reaches the second ssmmessages address when the first is black-holed #685: hosts order [AZ1, AZ2], AZ1 black-holed; default ControlChannelDialTimeoutMillis of 20000):
  • Detection (Reach the retry limit) at T0 + 11.6 s. This change does not touch detection; the delay depends on where the black-hole falls in the 54 s ping cycle (45 s in fault2).
  • The websocket dial abandoned AZ1 and opened to 172.20.5.11:443 10.1 s after it started (unchanged).
  • The token request then dialed AZ1 first, as before, but abandoned it after 9.997 s (4 SYNs at +0, +1.0, +3.0, +7.0 s; no 5th) instead of 15 s, and dialed AZ2. On AZ2 the request was written 43 ms after the SYN and answered 63 ms later.
  • SSM Connection channel status is set to ssmmessages at T0 + 32.4 s, 20.8 s after detection. In CreateControlChannel request never reaches the second ssmmessages address when the first is black-holed #685 the same fault never recovered in 10 minutes.
  • On AZ2 the token request sent application data and received the response (3182 B sent, 7066 B received). In fault2 that connection carried only a TLS handshake and no request.
  • Recovered on the first attempt: the websocket was opened once, and ssm-agent-worker.log has no ERROR and no Client.Timeout exceeded lines for the whole run.
  • The connection established on that attempt stayed up for the rest of the fault and after the black-hole was removed; no new connection to AZ1 was opened.
  • Run Command was not sent during the fault (same procedure as fault2). Commands sent 4 s after the black-hole was removed succeeded over that same AZ2 connection.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.