fix(email): explain the health and probe mismatch

.github/workflows/read-email-probe-state.yaml

@@ -0,0 +1,92 @@
+name: Read Email Probe State
+
+on:
+  workflow_dispatch:
+    inputs:
+      confirm:
+        description: "Type exactly: read-email-probe-state"
+        required: true
+        type: string
+
+permissions: {}
+concurrency:
+  group: prod-deploy
+  cancel-in-progress: false
+  queue: max
+
+jobs:
+  observe:
+    name: Compare existing Email probe and access state
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    environment: prod
+    permissions:
+      contents: read
+    env:
+      MAIL_ACCESS_CONFIRM: ${{ inputs.confirm }}
+    steps:
+      - name: Require the confirmed first-attempt main dispatch
+        shell: bash
+        run: |
+          set -euo pipefail
+          [[ "${GITHUB_REPOSITORY}" == devantler-tech/platform &&
+             "${GITHUB_EVENT_NAME}" == workflow_dispatch &&
+             "${GITHUB_REF}" == refs/heads/main &&
+             "${GITHUB_RUN_ATTEMPT}" == 1 &&
+             "${MAIL_ACCESS_CONFIRM}" == read-email-probe-state ]]
+
+      - name: Checkout the reviewed dispatch revision
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+
+      - name: Set up the repository Go toolchain
+        uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
+        with:
+          go-version-file: go.mod
+          cache: false
+
+      - name: Build the reviewed observer before restoring credentials
+        shell: bash
+        run: go build -o "$RUNNER_TEMP/observe-email-startup" ./scripts/diagnose-email-startup
+
+      - name: Install the repository-approved Kubernetes client
+        shell: bash
+        env:
+          KUBECTL_VERSION: v1.36.2
+          KUBECTL_SHA256: 1e9045ec32bea85da43de85f0065358529ea7c7a152eca78154fba5b58c27d82
+        run: |
+          set -euo pipefail
+          kubectl_path="$RUNNER_TEMP/email-observation-kubectl"
+          curl --disable --fail --silent --show-error --location --proto '=https' --proto-redir '=https' --max-redirs 3 --connect-timeout 10 --max-time 60 \
+            "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o "$kubectl_path"
+          printf '%s  %s\n' "$KUBECTL_SHA256" "$kubectl_path" | sha256sum --check
+          mkdir -p "$RUNNER_TEMP/email-observation-tools"
+          install -m 0755 "$kubectl_path" "$RUNNER_TEMP/email-observation-tools/kubectl"
+          printf '%s\n' "$RUNNER_TEMP/email-observation-tools" >> "$GITHUB_PATH"
+
+      - name: Observe the existing serving container
+        shell: bash
+        env:
+          KUBE_CONFIG: ${{ secrets.KUBE_CONFIG }}
+          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
+        run: |
+          set +x
+          set -euo pipefail
+          umask 077
+          if [[ -z "$KUBE_CONFIG" || ${#KUBE_CONFIG} -gt 1048576 ]]; then
+            printf '%s\n' 'MAIL_PROBE_STATE=INCONCLUSIVE'
+            exit 3
+          fi
+          proof_dir="$(mktemp -d "$RUNNER_TEMP/email-startup-context.XXXXXX")"
+          trap 'rm -rf "$proof_dir"' EXIT
+          export KUBECONFIG="$proof_dir/kubeconfig"
+          printf '%s' "$KUBE_CONFIG" > "$KUBECONFIG"
+          unset KUBE_CONFIG
+          if ! ./scripts/use-prod-stable-api-endpoint.sh >/dev/null; then
+            printf '%s\n' 'MAIL_PROBE_STATE=INCONCLUSIVE'
+            exit 3
+          fi 2>"$proof_dir/endpoint.stderr"
+          unset HCLOUD_TOKEN
+          "$RUNNER_TEMP/observe-email-startup"

docs/email-startup-observation.md

@@ -51,3 +51,57 @@ Native Linux regression coverage acquires only an owned temporary POSIX lock and
 checks its real kernel identity. Other tests exercise the actual embedded probe
 and workflow shell steps with synthetic files and credentials, including
 replacement identities, finite waiting and private-output refusal.
+
+## Compare health requests and stored access rules
+
+**Read Email Probe State** is a separate opt-in workflow. Dispatch it from reviewed
+`main`, first attempt only, with `read-email-probe-state`. It shares the production
+deployment lock and retains the fixed serving image, command, mount and runtime
+binding. The serving container must have no environment entries or imports,
+preventing a shell startup hook from observing its stdin.
+
+This observer sends the existing one-second HTTP/1.1 health requests, with the
+native Kubernetes 1.36 `kube-probe/1.36` User-Agent, to loopback and the bound Pod address. The bound
+node must run a supported stable 1.36 kubelet before and after the observation;
+a changed node identity or version is inconclusive. The header follows the
+kubelet minor, independently of the Kubernetes client version. These requests
+originate inside the Pod; they do not reproduce a kubelet's actual network path.
+
+Only after verifying the target does the protected workflow read the two named
+settings and administrator Secrets. The administrator input retains its existing
+`credential` and `password` keys; only `password` is extracted for authentication.
+Missing or extra keys are inconclusive. It authenticates the fixed loopback API with
+the existing administrator, requesting only allowed/blocked address and expiry
+fields. Credentials travel in memory and stdin, never command arguments,
+exported environment, files or artifacts. Native stdout/stderr is captured
+privately; public output contains only the classifications below. The observer
+never changes access rules or application data.
+
+The request queries both address inventories with totals and a fixed limit,
+then fetches only those IDs. The decoder refuses incomplete pages, totals over
+256, missing or duplicate IDs, failed methods, missing objects, unrequested
+fields, invalid expiry data and responses over 64 KiB. A bare get-all call
+silently stops at the server's configured get limit, so it is insufficient proof.
+Pod identity, targets, public configuration and both Secret identities/versions
+are checked again after the single exec. An unsuccessful exec is never retried.
+
+Public output contains `MAIL_PROBE_PATHS` (`ALL_LOCAL_TARGETS_HTTP_200`,
+`LOOPBACK_PATH_FAILED` or `POD_ADDRESS_PATH_FAILED`), `MAIL_ACCESS_STATE`
+(`STORED_HOST_BLOCK_MATCH`, `STORED_HOST_ALLOW_OVERRIDE`,
+`NO_ACTIVE_STORED_BLOCKS` or `OTHER_ACTIVE_STORED_BLOCKS`) and
+`MAIL_KUBERNETES_READY` (`TRUE` or `FALSE`). A guard or collection failure emits
+only `MAIL_PROBE_STATE=INCONCLUSIVE` and exits three. An observed problem exits
+one; a ready sample with successful targets and no active stored blocks exits zero.
+
+The host address is a candidate from bound Pod status. A stored rule match does
+not establish the source of a failed kubelet connection or attest the application's
+in-memory security cache. The API read is a bounded observation, not an atomic
+transaction over concurrently changing application state. Allow rules override
+matching blocks, and expired entries are excluded. No classification establishes
+sustained recovery or authorizes clearing a block.
+
+The native response fixture was captured from checksum-verified Stalwart 0.16.27
+with a synthetic store and a deliberately blocked documentation address.
+Regressions exercise the actual embedded shell over HTTP, complete native
+query/get schemas, expiry/override precedence, credential framing, dispatch
+guards and changed runtime/input identities. Production requires a bound result.