Repository navigation
Private mail server stalls on startup checks since 01:38 UTC #5413
Description
🤖 Generated by the Agentic Engineer
Evidence
The company mail service has been unavailable since 2026-10-27 01:38 UTC. It had been running normally since it started on 2026-10-26 16:50 UTC.
- At 01:38 UTC the server stopped answering its liveness check, so Kubernetes restarted it.
- Since then, every restart fails its 120-second startup check. The management port accepts a connection and then drops it ("connection reset by peer", "EOF", "server closed idle connection"). About 10 restarts so far.
- While it waits, the process does almost nothing (about 1m CPU and 57 MiB memory) and writes no logs, because no log output is configured.
- The configuration step that runs first still passes:
Private mail configuration verified; recovery process stopped. - The data volume is attached and healthy, and only about 7% full. The node is healthy. The platform's own Flux Kustomizations are Ready.
Audience / problem
The maintainer, and anything waiting on company Email (#5030), currently gets no mail service. The service also writes no logs, so the next person who looks has nothing to diagnose it from.
Hypothesis
The server is waiting on something it cannot reach or acquire, so its health endpoint never answers. Likely candidates are an outbound fetch at startup that the namespace's default-deny egress blocks, or a store lock left behind by the liveness kill. If the server logs to stdout, the next restart will show which one it is.
Success signal
- The
private-mailStatefulSet is back to 1/1 Ready and stays Ready for 24 hours with no restarts. - The service writes logs to stdout, so a future stall can be diagnosed from
kubectl logs.
Smallest useful change
- Turn on stdout logging in the server configuration. Restarts already happen every two minutes, so the next one will show what the server is waiting for.
- Fix that root cause, whether it is an egress allowance or a startup or lock problem. Do not lengthen the probes to hide it.
Acceptance criteria
- Logs on stdout show why the server stalled.
- Root cause fixed in Git, validated statically, and reconciled by Flux.
-
platform-live-health.shno longer reportsprivate-mail-0.
Reacted by coderabbitai
Activity
coderabbitai commentedon Oct 27, 2026
🔗 Related PRs
#5335 - fix(mail): separate recovery from serving management [merged]
📝 Issue Planner
Check the box below or use the
@coderabbitai plancommand to generate an implementation plan and prompts that you can use with your favorite coding assistant.🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:💬 Have feedback or questions? Drop into our discord!
devantler commentedon Oct 27, 2026
Duplicate of #5412: both track the same Email outage beginning at 01:38 UTC. The repair is claimed on #5412, where root-cause diagnosis, safe diagnostics, data preservation and stable deployed availability remain required.
The initializer's recorded successful completion predates this outage; it is not a fresh recovery result. No probe weakening or live configuration mutation has been performed.