BUG: K8s provider cert mismatch (x509) when tofu plan/apply uses SSH tunnel

[jgruberf5]

Summary

When the OpenTofu K8s tunnel (tofu_k8s_tunnel.py) opens an SSH tunnel for tofu plan/tofu apply, it writes a provider_override.tf that points the kubernetes provider at https://127.0.0.1:{tunnel_port}. However, the EKS API server certificate is only valid for the real endpoint hostname (e.g. *.eks.amazonaws.com) and internal K8s service IPs — not 127.0.0.1. This causes every kubernetes_* resource refresh to fail with:

x509: certificate is valid for 172.20.0.1, 10.0.20.202, 172.16.45.87, not 127.0.0.1

This blocks all plan/apply operations on modules that use the kubernetes provider through SSH tunnels.

Root Cause

In backend/services/tofu_k8s_tunnel.py, the _write_provider_override() function writes:

provider "kubernetes" {
  host                   = "https://127.0.0.1:{local_port}"
  cluster_ca_certificate = base64decode(...)
  token                  = ...
}

The host is 127.0.0.1 but TLS verification checks the certificate's SANs against the connection hostname, which fails.

Note: The Forge API/UI K8s client (cluster_utils.py, kubernetes/_base.py) handles this correctly by setting insecure-skip-tls-verify on the kubeconfig. But the Terraform provider override does not have an equivalent.

Fix

Add tls_server_name to the provider override. The Terraform kubernetes provider supports this attribute — it sets the TLS SNI header so cert verification checks against the original hostname instead of 127.0.0.1:

def _write_provider_override(work_dir, local_port, eks_endpoint):
    from urllib.parse import urlparse
    _parsed_ep = urlparse(eks_endpoint)
    tls_server_name = _parsed_ep.hostname or ""

    override_content = f'''
provider "kubernetes" {{
  host                   = "https://127.0.0.1:{local_port}"
  tls_server_name        = "{tls_server_name}"
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}}
'''

The eks_endpoint is already passed to the function (currently only used in a comment). This parses the hostname and uses it for TLS SNI.

Status

• Fix applied on server (10.145.33.194) in ~/git/bnk-forge/backend/services/tofu_k8s_tunnel.py, containers rebuilt — plan/apply works through SSH tunnel
• Not committed to git — needs PR
• Should also consider adding tls_server_name for the helm provider if/when helm provider overrides are added

Impact

Without this fix, any module with kubernetes provider resources (e.g. high-performance-nodes with multus/sriov/dpdk k8s manifests) cannot plan or apply when the cluster is accessed via SSH tunnel.

---

_{Migrated from sp-prod-field/bnk-forge #22 (opened 2026-04-14; original labels: bug). That repository is archived and read-only.
Bare #NNN references in the text above refer to issues and PRs in the original repository, not to numbering here.}

[jgruberf5]

Investigated against current staging — the reported code path no longer exists, and the x509 failure it describes is not reachable.

backend/services/tofu_k8s_tunnel.py is not in this repository and never has been (git log --all -- '*tofu_k8s_tunnel*' is empty), and nothing writes a provider_override.tf any more. The fix described here was applied directly on server 10.145.33.194 and never committed, so it did not survive into the public repo — but neither did the code it patched.

What replaced it. Tunnelled clusters are now handled in backend/services/execution/config_writer.py, which rewrites the kubeconfig rather than generating a provider override:

c["cluster"]["server"] = f"https://127.0.0.1:{tunnel_port}"
c["cluster"]["insecure-skip-tls-verify"] = True
c["cluster"].pop("certificate-authority-data", None)

The Terraform kubernetes provider reads that kubeconfig, so it never verifies the cert against 127.0.0.1 and the x509: certificate is valid for ... not 127.0.0.1 error cannot occur. That is the same approach this issue notes the Forge API/UI client already took (cluster_utils, kubernetes/_base.py).

Suggest closing — but flagging one thing, because the current fix is weaker than the one proposed here:

insecure-skip-tls-verify doesn't just fix the hostname mismatch, it disables verification entirely, so a tunnelled plan/apply has no protection against a MITM on the tunnel path. The tls_server_name approach suggested above preserves verification by checking against the original hostname; kubeconfig has the equivalent tls-server-name field in the cluster block, so the same trick applies:

server: https://127.0.0.1:<port>
tls-server-name: <original-eks-hostname>
# certificate-authority-data retained rather than stripped

That is a hardening change rather than a bug fix, and it touches every tunnelled cluster, so I have not bundled it here. Worth its own issue if it's wanted — happy to open one.

_{Found while working through the severity:high bug queue; no PR opened since there is nothing left to fix.}

[jgruberf5]

Fixed in #150, squash-merged to staging as b5fb17bf.

Closing manually — the auto-close Action does not fire on these PRs because the PR template's Fixes / Implements: #N format is not a shape its parser accepts.