Description
[Bug] Docker containers never restart when Java process crashes — broken cron monitor + tail -f /dev/null supervision
Before submit
- I have confirmed and searched that there are no similar problems in the historical issue and documents
Environment
- HugeGraph version: 1.7.0+
- Deployment: Docker / docker-compose
- Related PR: perf(docker): improve all images build cache efficiency #3025
Expected & Actual behavior
Expected:
When the HugeGraph Java process crashes inside a Docker container, the container should exit and Docker's restart policy (restart: unless-stopped) should automatically bring it back up.
Actual:
The container stays in Up state permanently even after Java crashes. docker ps shows green. Users get Connection refused. The container never restarts on its own. Manual intervention is required every time.
# Java crashes at T=0:30
$ docker ps
CONTAINER ID IMAGE STATUS PORTS
abc123 hugegraph/server Up 2 hours 0.0.0.0:8080->8080/tcp
# Container looks healthy but:
$ curl http://localhost:8080/versions
curl: (7) Failed to connect to localhost port 8080: Connection refused
Root Cause Analysis
There are three compounding problems:
1. crond is never started — the watchdog is completely dead
cron is installed in all four Dockerfiles but dumb-init only launches docker-entrypoint.sh. Nobody starts crond. So even if start-hugegraph.sh -m true were called, start-monitor.sh registers the crontab job but since crond is not running, monitor-hugegraph.sh never fires. The entire watchdog silently does nothing in containers.
What happens on a VM: What happens in Docker:
crond reads crontab every crond is NOT running
minute monitor-hugegraph.sh NEVER fires
monitor-hugegraph.sh fires HugeGraph stays dead forever
HugeGraph gets restarted
2. tail -f /dev/null means zero supervision
All three docker-entrypoint.sh files background the Java process then sleep forever:
# hugegraph-server entrypoint (current)
./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
# ... post-startup checks ...
tail -f /dev/null # ← keeps container alive with NO watchdogWhen Java crashes, tail -f /dev/null keeps running. The container never exits. Docker's restart: unless-stopped only triggers on container exit — since the container never exits, the restart policy never fires. The container stays Up (unhealthy) forever.
3. HEALTHCHECK only exists in docker-compose.yml, not in the Dockerfiles
Health checks are defined per service in docker-compose.yml but none of the four Dockerfiles have a HEALTHCHECK instruction. So docker run without compose has no health reporting at all. depends_on: condition: service_healthy only works because compose injects the check at runtime — it is not baked into the image.
4. Foreground mode is broken in start-hugegraph.sh
start-hugegraph.sh has a -d false foreground flag but it is broken. $!, pid file write, trap, wait_for_startup, disown, and OPEN_MONITOR all run unconditionally after the daemon/foreground if/else block — meaning in foreground mode they all execute after Java has already exited, with empty/stale values. Java's exit code is lost and the script always exits 0.
5. No foreground mode exists at all in start-hugegraph-pd.sh and start-hugegraph-store.sh
Both scripts always background Java unconditionally with exec java ... & regardless of any flag. There is no -d flag and no foreground path.
Impact
CURRENT — Java crashes inside container:
T=0:30 Java crashes (OOM, segfault, deadlock, etc.)
T=0:30 tail -f /dev/null keeps running
T=0:30 Container stays "Up" — Docker sees nothing wrong
T=1:00 HEALTHCHECK marks container "unhealthy" (compose only)
T=∞ Container stays unhealthy forever, never restarts
docker ps shows: Up 2 hours (unhealthy)
Users get: Connection refused
AFTER FIX — Java crashes inside container:
T=0:30 Java crashes
T=0:30 Entrypoint exits → dumb-init exits → container exits
T=0:30 Docker restart policy fires immediately
T=0:31 New container starts
T=1:41 docker ps shows: Up 1 min (healthy)
Additional bug found during investigation
The shipped default conf/rest-server.properties has:
restserver.url=127.0.0.1:8080
No http:// scheme. On macOS, curl fails immediately with "Protocol not supported" causing wait_for_startup to always time out and start-hugegraph.sh to exit 1 even though the server starts fine. Every other config in the repo uses http:// explicitly — raft CI configs, the Dockerfile sed patch, cluster test templates, and the Java ServerOptions default. The shipped default is inconsistent and breaks local macOS development.
Related
- Follow-up to perf(docker): improve all images build cache efficiency #3025
- Deferred from #3025 discussion (comment by @imbajin ): "Either drop cron or start it in docker-entrypoint.sh" → decision was to keep cron in perf(docker): improve all images build cache efficiency #3025 (comment) and fix properly in a follow-up
Related Issues
| Issue | Description |
|---|---|
| #3054 | CI test skip exit code — non-blocking follow-up flagged in reviews of #3044 and #3047 |
Implementation Plan & Progress
Phase 1: Bash Script Foreground Support (Completed) ✅
- Chunk 1: Fix
start-hugegraph.shforeground mode andrestserver.urlconfig scheme — Merged in [fix(server): fix foreground mode exit code propagation in startup scripts #3044] - Chunk 2: Add
-dflag and foreground mode tostart-hugegraph-pd.sh— Merged in [fix(pd,store): fg mode exit code propagation in startup scripts #3047] - Chunk 3: Add
-dflag and foreground mode tostart-hugegraph-store.sh— Merged in [fix(pd,store): fg mode exit code propagation in startup scripts #3047]
Phase 2: Docker Phase (Completed) ✅
- Chunk 4: Update
docker-entrypoint.shfor PD — replacetail -f /dev/nullwith-d false— Merged in [fix(docker): supervise Java process in entrypoints instead of tail -f /dev/null #3051] - Chunk 5: Update
docker-entrypoint.shfor Store — replacetail -f /dev/nullwith-d false— Merged in [fix(docker): supervise Java process in entrypoints instead of tail -f /dev/null #3051] - Chunk 6: Update
docker-entrypoint.shfor Server — replacetail -f /dev/nullwithtail --pidsupervision + SIGTERM trap — Merged in [fix(docker): supervise Java process in entrypoints instead of tail -f /dev/null #3051] - Chunk 7: Add native
HEALTHCHECKinstructions to all four Dockerfiles — Merged in [chore(docker): add HEALTHCHECK & clean Dockerfiles #3052] - Chunk 8: Remove unused
cronfrom Dockerfileapt-getinstallations — Merged in [chore(docker): add HEALTHCHECK & clean Dockerfiles #3052]
Phase 3: Documentation & CI Hardening
- Chunk 11: Fix CI test skip exit code — exit
77instead of0when required tools are missing, so CI can distinguish "skipped" from "passed" — tracked in ci(test): distinguish "skipped due to missing tools" from "tested and passed" in startup script tests #3054, PR chore(ci): exit 77 when tools missing to distinguish skip from pass #3055 - Chunk 10: Update documentation to explain Docker-native process supervision and the
-d true|falseflag — PRs open: docs: document -d flag and Docker process supervision model #3056 (in-repo READMEs) and apache/hugegraph-doc#461 (EN + CN guides)
Activity
dosubot commentedon May 30, 2026
This is a confirmed architectural issue in HugeGraph's Docker setup. The root cause is clear from the entrypoint scripts:
All three components (Server, Store, PD) use
tail -f /dev/nullas the final command in their entrypoint scripts, keeping the container's PID 1 alive indefinitely even after Java crashes [1] [2] [3].The startup scripts (
start-hugegraph-store.sh,start-hugegraph-pd.sh) background the Java process with&[4] [5], so even thoughexecis used, the shell continues running since&prevents the process replacement. Docker sees PID 1 (tail -f /dev/null) as alive → container stays "Up" → restart policies never trigger.Workaround until this is fixed upstream: Add a
healthcheckto yourdocker-compose.ymland combine it withrestart: unless-stopped:Note: Docker's restart policy alone won't help here since the container never exits. The healthcheck will mark it unhealthy, but you'd need an external tool (like
autoheal) to actually restart unhealthy containers, or use Docker Swarm/Kubernetes which natively restart unhealthy containers.Proper fix would involve modifying the entrypoint scripts to:
tail -f /dev/null&) so the container exits when Java diesdumb-init(already installed in Store/PD images) for signal handlingThere's also an unused cron-based monitor in the Server component (enabled via
-mflag instart-hugegraph.sh), but it's not activated in the Docker entrypoint and only covers Server — not Store or PD.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
bitflicker64 commentedon May 30, 2026
Implementation Plan
Breaking the work into independently reviewable and testable chunks. Each chunk has its own test script, can be reviewed in isolation, and builds on the previous one.
Chunk 1 — Fix
start-hugegraph.shforeground modeFile:
hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.shProblem: All post-branch logic (
PID="$!", pid file write,trap,wait_for_startup,disown,OPEN_MONITOR) runs unconditionally after the daemon/foreground if/else block. In foreground mode the script blocks athugegraph-server.shuntil Java exits, then all those lines run with$!empty/stale,wait_for_startupimmediately fails (Java already dead), and the script exits 0 losing Java's exit code entirely.Fix: Move all post-branch logic inside the
DAEMON == "true"branch. Addexit $?to the foreground branch so Java's exit code propagates out.Test:
hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph.shChunk 2 — Add
-dflag tostart-hugegraph-pd.shFile:
hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.shProblem: No foreground mode exists — Java is always backgrounded unconditionally with
exec java ... &. No-dflag.wait_for_startupexists in pd'sutil.shbut is never called — daemon startup has no health polling at all, callers compensate with hardcodedsleep 10.Fix: Add
DAEMON="true"default (consistent with existing style in this file), add-dtogetopts, split theexec javablock into daemon/foreground branches, wire upwait_for_startupin daemon mode, addexit $?to foreground branch.Test:
hugegraph-pd/hg-pd-dist/src/assembly/travis/test-start-hugegraph-pd.shChunk 3 — Add
-dflag tostart-hugegraph-store.shFile:
hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.shProblem: Identical to chunk 2 — no foreground mode, always backgrounds Java unconditionally.
Fix: Identical structure to chunk 2.
Test:
hugegraph-store/hg-store-dist/src/assembly/travis/test-start-hugegraph-store.shChunk 4 — Fix pd entrypoint
File:
hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.shProblem:
Fix: Pass
-d falsenow that pd has foreground mode. Removetail -f /dev/null. When Java exits the script exits, container exits, restart policy fires.Chunk 5 — Fix store entrypoint
File:
hugegraph-store/hg-store-dist/docker/docker-entrypoint.shProblem: Same as chunk 4.
Fix: Same as chunk 4 but for store.
Chunk 6 — Fix server entrypoint
File:
hugegraph-server/hugegraph-dist/docker/docker-entrypoint.shProblem:
Why
-d falsealone doesn't work here:wait-partition.shmust run after the server is already up in the background. The structure must stay: start in background → post-startup checks → then supervise.Fix: Keep the background start and all post-startup checks unchanged. Replace only
tail -f /dev/nullwithwait $(cat ./bin/pid). When Java exitswaitreturns immediately, entrypoint exits with non-zero, container exits, restart policy fires.Chunk 7 — Add
HEALTHCHECKto all four DockerfilesFiles:
hugegraph-server/Dockerfile,hugegraph-server/Dockerfile-hstore,hugegraph-pd/Dockerfile,hugegraph-store/DockerfileProblem: No
HEALTHCHECKin any Dockerfile. Health checks only exist indocker-compose.yml. Plaindocker runhas no health reporting.depends_on: condition: service_healthyrelies on compose injecting the check at runtime rather than it being baked into the image.Fix: Add
HEALTHCHECKto each Dockerfile beforeENTRYPOINT, mirroring the exact values already indocker-compose.yml:Chunk 8 — Remove
cronfrom all four DockerfilesFiles: All four Dockerfiles
Problem:
cronis installed butcrondis never started — it has no purpose in the image now that supervision is Docker-native.Fix: Remove
cronfrom theapt-get installstep in all four Dockerfiles. Keepdumb-init,procps,curl,lsof.What stays untouched:
monitor-hugegraph.sh— preserved, still valid for VM deploymentsstart-monitor.sh— preserved, still valid for VM deploymentsdocker-compose.ymlhealthcheck definitions — already correct, no changes neededChunk 9 — Fix default
restserver.urlschemeFile:
hugegraph-server/hugegraph-dist/src/assembly/static/conf/rest-server.propertiesProblem: Shipped default is
restserver.url=127.0.0.1:8080— nohttp://scheme.wait_for_startuppasses this raw value to curl. On macOS, curl fails with "Protocol not supported" causingstart-hugegraph.shto always exit 1 on local development even though the server starts correctly. On Linux, curl happens to handle scheme-less URLs by defaulting to HTTP — masking the bug.Every other config in the repo uses
http://explicitly: raft CI configs, the Dockerfilesedpatch, cluster test templates, and JavaServerOptionsdefault. The shipped default is the only inconsistency.Fix: Change line 3 of
rest-server.propertiesfrom127.0.0.1:8080tohttp://127.0.0.1:8080.Chunk 10 — Docs
Update relevant README or
docker/docs to explain:restart: unless-stoppedHEALTHCHECKis now baked into the Dockerfiles sodocker psreports health without needing composestart-hugegraph.sh -m true(cron-based monitor) is for VM/bare-metal deployments only — unchanged and fully functionalBlast radius
All existing callers of
start-hugegraph.sh,start-hugegraph-pd.sh, andstart-hugegraph-store.shpass no-dflag and rely on the defaultDAEMON="true". Daemon mode behavior is completely unchanged. Only new callers that explicitly pass-d false(the Docker entrypoints after this PR) are affected by chunks 1-3.bitflicker64 commentedon May 30, 2026
@dosu what do u think about the implementation plan
dosubot commentedon May 30, 2026
The plan is well-structured and technically sound. A few observations after verifying against the current code:
Chunk 1 — Looks correct. The server's
start-hugegraph.shalready has aDAEMONvariable and-dflag [1], but the post-branch logic (PID="$!", trap,wait_for_startup,disown, monitor) does run unconditionally after the if/else block [2]. Moving that into the daemon branch and addingexit $?to the foreground branch is the right fix.Chunks 2 & 3 — Confirmed. PD and Store both unconditionally background Java with
&and have no-dflag [3] [4]. Adding a daemon/foreground split mirrors the pattern already established in the server script.Chunk 6 — One nuance worth noting. The plan uses
wait "$(cat ./bin/pid)"after the post-startup checks. This works becausestart-hugegraph.shwrites the PID to$BIN/pid[5]. However, sincedumb-initis theENTRYPOINTin PD/Store Dockerfiles [6] but not in the server Dockerfile, you'll want to confirm thatwaiton a child PID works correctly under the shell that runs the entrypoint. If the server Dockerfile doesn't usedumb-init, the backgrounded Java process is a direct child of the entrypoint bash shell, sowait $PIDshould work fine. Just noting it's a difference from the PD/Store containers.Chunk 8 — Confirmed. Both PD and Store Dockerfiles do install
cronvia apt-get [7] [8], and the service is started during init. Removing it is clean since supervision is now Docker-native.Chunk 9 — Confirmed. The shipped default is indeed
127.0.0.1:8080with no scheme [9]. Addinghttp://aligns it with every other config in the repo.Overall assessment: The blast radius analysis is accurate — all existing callers use the default
DAEMON="true"so daemon mode is untouched. The chunking is well-sequenced (scripts first, then entrypoints that depend on them, then Dockerfiles, then docs). Each chunk is independently testable.One suggestion: in Chunk 6, consider adding a comment in the entrypoint explaining why the server uses the
start-in-background → post-checks → wait $PIDpattern instead of-d falselike PD/Store, since future maintainers might otherwise "simplify" it and breakwait-partition.sh.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
imbajin commentedon May 30, 2026
Thanks for opening this. I think this is a valid follow-up to #3025, not something already fixed by it.
#3025 intentionally kept runtime behavior unchanged and focused on Docker build/cache hygiene. This issue is about a different layer: Docker runtime lifecycle and process supervision.
Core Context
The main problem is that the Java process lifecycle is currently decoupled from the container lifecycle.
At the moment, the Docker entrypoints start the HugeGraph Java process in the background, then keep the container alive with
tail -f /dev/null.That means when Java dies, Docker still sees the container as running, so
restart: unless-stoppeddoes not trigger.Expected Docker-native behavior should be closer to:
Conclusion
I agree this should be treated as a real Docker runtime lifecycle bug.
The key points are:
tail -f /dev/nullhides Java process failure from Docker.healthcheckhelps observability, but ordinary Docker does not restart a container just because it is unhealthy.wait-partition.sh; it likely needs the pattern: start in background, run post-start checks, then wait on the actual Java PID.Suggested Fix Direction
A clean fix should probably be split like this:
Fix startup script foreground semantics
start-hugegraph.sh -d falseso it propagates the Java exit code correctly.start-hugegraph-pd.sh.start-hugegraph-store.sh.Fix Docker entrypoints
tail -f /dev/null.Document the lifecycle model
HEALTHCHECKreports health.restart: unless-stoppedrestarts only after container exit.tail.Validation
The fix should include a runtime restart validation, not only a Docker build check.
Example validation flow:
Expected result after the fix:
Upwith a dead service.RestartCountincreases./versionsbecomes reachable again after restart.The same lifecycle check should be covered for Server, PD, and Store. For Server with hstore, we should also verify that the existing post-start partition wait logic is not skipped.
bitflicker64 commentedon May 30, 2026
@imbajin Thanks for the validation and the runtime restart test suggestion. I'll start with the startup script changes (chunks 1–3), get the foreground mode behavior and tests in place first, then move on to the Docker entrypoint/runtime lifecycle changes in a follow-up PR.
bitflicker64 commentedon May 31, 2026
hey @imbajin chunk 1 PR up: #3044
fixes foreground mode in start-hugegraph.sh, adds test script (all green on CI). chunks 2–3 next.
bitflicker64 commentedon Jun 4, 2026
hey @imbajin chunks 2–3 PR up: #3047
same foreground mode fix for start-hugegraph-pd.sh and start-hugegraph-store.sh, tests wired into pd-store-ci.yml.
fix(test): clean up logs dir in test cleanup to avoid RAT license fai…
fix(pd,store): fg mode exit code propagation in startup scripts (#3047)
bitflicker64 commentedon Jun 5, 2026
@imbajin chunk 4 PR up: #3051 replaces tail -f /dev/null in all three Docker entrypoints with proper Java process supervision. Also, not sure if you've been checking Discord recently , had a quick question about the Feishu access issue When you get a chance, could you take a quick look ?
bitflicker64 commentedon Jun 5, 2026
@imbajin chunk 7+8 PR up: #3052 ,adds healthchecks to all four Dockerfiles (server/PD/Store/hstore) and removes unused cron from the images.
ci(docker): verify HEALTHCHECK is present in built images
feat(docker): add HEALTHCHECK and remove cron from Dockerfile-hstore
chore(docker): add HEALTHCHECK & clean Dockerfiles (#3052)
bitflicker64 commentedon Jun 8, 2026
@imbajin — the core fix is complete. All merged PRs:
#3044 — fix foreground mode exit code propagation in start-hugegraph.sh
#3047 — same fix for start-hugegraph-pd.sh and start-hugegraph-store.sh
#3051 — replace tail -f /dev/null with proper Java process supervision in all three Docker entrypoints
#3052 — add HEALTHCHECK to all four Dockerfiles and remove unused cron from images
Also have a non-blocking CI hardening PR open (#3055) to distinguish "skipped" from "passed" in the startup test suite, flagged in your reviews of #3044 and #3047.
Starting on the documentation PR (Chunk 10) next — will cover the -d true|false flag, Docker-native supervision model, and HEALTHCHECK behavior.
Is there anything else you'd like addressed before we close this out?
ci(test): use preflight/guard pattern to show skipped steps as grey
bitflicker64 commentedon Jun 9, 2026
@imbajin Both docs PRs are now open — Chunk 10 complete:
Also open: chore(ci): exit 77 when tools missing to distinguish skip from pass #3055 (CI test skip exit code — Chunk 11), pending review.
Once #3055, #3056, and the hugegraph-doc PR merge, this issue can be closed.
chore(ci): exit 77 when tools missing to distinguish skip from pass (#…
docs: document -d flag and Docker process supervision model (#3056)
docs: add Docker process supervision model and -d flag (#461)
docs: add Docker process supervision model and -d flag (#461)
bitflicker64 commentedon Jun 11, 2026
@imbajin I think this issue can be safely closed, unless there's something I'm overlooking, of course.
Also, is there anything else you'd like me to take care of?
I had a couple of questions as well. Would Discord still be the best place to ask them, or would you prefer that I send an email instead?
imbajin commentedon Jun 12, 2026
thanks & lgtm now
for questions: both fine for me (discord and mail)
task: align issue format with apache#3043, rename to [Improve]