Skip to content

[Bug] Docker containers never restart when Java process crashes — broken cron monitor + tail -f /dev/null supervision #3043

Closed

Description

@bitflicker64

[Bug] Docker containers never restart when Java process crashes — broken cron monitor + tail -f /dev/null supervision

Before submit

  • I have confirmed and searched that there are no similar problems in the historical issue and documents

Environment


Expected & Actual behavior

Expected:
When the HugeGraph Java process crashes inside a Docker container, the container should exit and Docker's restart policy (restart: unless-stopped) should automatically bring it back up.

Actual:
The container stays in Up state permanently even after Java crashes. docker ps shows green. Users get Connection refused. The container never restarts on its own. Manual intervention is required every time.

# Java crashes at T=0:30
$ docker ps
CONTAINER ID   IMAGE              STATUS               PORTS
abc123         hugegraph/server   Up 2 hours           0.0.0.0:8080->8080/tcp

# Container looks healthy but:
$ curl http://localhost:8080/versions
curl: (7) Failed to connect to localhost port 8080: Connection refused

Root Cause Analysis

There are three compounding problems:

1. crond is never started — the watchdog is completely dead

cron is installed in all four Dockerfiles but dumb-init only launches docker-entrypoint.sh. Nobody starts crond. So even if start-hugegraph.sh -m true were called, start-monitor.sh registers the crontab job but since crond is not running, monitor-hugegraph.sh never fires. The entire watchdog silently does nothing in containers.

What happens on a VM:          What happens in Docker:
  crond reads crontab every      crond is NOT running
  minute                         monitor-hugegraph.sh NEVER fires
  monitor-hugegraph.sh fires     HugeGraph stays dead forever
  HugeGraph gets restarted

2. tail -f /dev/null means zero supervision

All three docker-entrypoint.sh files background the Java process then sleep forever:

# hugegraph-server entrypoint (current)
./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
# ... post-startup checks ...
tail -f /dev/null   # ← keeps container alive with NO watchdog

When Java crashes, tail -f /dev/null keeps running. The container never exits. Docker's restart: unless-stopped only triggers on container exit — since the container never exits, the restart policy never fires. The container stays Up (unhealthy) forever.

3. HEALTHCHECK only exists in docker-compose.yml, not in the Dockerfiles

Health checks are defined per service in docker-compose.yml but none of the four Dockerfiles have a HEALTHCHECK instruction. So docker run without compose has no health reporting at all. depends_on: condition: service_healthy only works because compose injects the check at runtime — it is not baked into the image.

4. Foreground mode is broken in start-hugegraph.sh

start-hugegraph.sh has a -d false foreground flag but it is broken. $!, pid file write, trap, wait_for_startup, disown, and OPEN_MONITOR all run unconditionally after the daemon/foreground if/else block — meaning in foreground mode they all execute after Java has already exited, with empty/stale values. Java's exit code is lost and the script always exits 0.

5. No foreground mode exists at all in start-hugegraph-pd.sh and start-hugegraph-store.sh

Both scripts always background Java unconditionally with exec java ... & regardless of any flag. There is no -d flag and no foreground path.


Impact

CURRENT — Java crashes inside container:
  T=0:30  Java crashes (OOM, segfault, deadlock, etc.)
  T=0:30  tail -f /dev/null keeps running
  T=0:30  Container stays "Up" — Docker sees nothing wrong
  T=1:00  HEALTHCHECK marks container "unhealthy" (compose only)
  T=∞     Container stays unhealthy forever, never restarts
          docker ps shows: Up 2 hours (unhealthy)
          Users get: Connection refused

AFTER FIX — Java crashes inside container:
  T=0:30  Java crashes
  T=0:30  Entrypoint exits → dumb-init exits → container exits
  T=0:30  Docker restart policy fires immediately
  T=0:31  New container starts
  T=1:41  docker ps shows: Up 1 min (healthy)

Additional bug found during investigation

The shipped default conf/rest-server.properties has:

restserver.url=127.0.0.1:8080

No http:// scheme. On macOS, curl fails immediately with "Protocol not supported" causing wait_for_startup to always time out and start-hugegraph.sh to exit 1 even though the server starts fine. Every other config in the repo uses http:// explicitly — raft CI configs, the Dockerfile sed patch, cluster test templates, and the Java ServerOptions default. The shipped default is inconsistent and breaks local macOS development.


Related

Related Issues

Issue Description
#3054 CI test skip exit code — non-blocking follow-up flagged in reviews of #3044 and #3047

Implementation Plan & Progress

Phase 1: Bash Script Foreground Support (Completed) ✅

Phase 2: Docker Phase (Completed) ✅

Phase 3: Documentation & CI Hardening

Activity

  1. dosubot commented on May 30, 2026

    @dosubot

    This is a confirmed architectural issue in HugeGraph's Docker setup. The root cause is clear from the entrypoint scripts:

    All three components (Server, Store, PD) use tail -f /dev/null as the final command in their entrypoint scripts, keeping the container's PID 1 alive indefinitely even after Java crashes [1] [2] [3].

    The startup scripts (start-hugegraph-store.sh, start-hugegraph-pd.sh) background the Java process with & [4] [5], so even though exec is used, the shell continues running since & prevents the process replacement. Docker sees PID 1 (tail -f /dev/null) as alive → container stays "Up" → restart policies never trigger.

    Workaround until this is fixed upstream: Add a healthcheck to your docker-compose.yml and combine it with restart: unless-stopped:

    services:
      hugegraph-server:
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:8080/versions"]
          interval: 15s
          timeout: 5s
          retries: 3
          start_period: 30s
        restart: unless-stopped

    Note: Docker's restart policy alone won't help here since the container never exits. The healthcheck will mark it unhealthy, but you'd need an external tool (like autoheal) to actually restart unhealthy containers, or use Docker Swarm/Kubernetes which natively restart unhealthy containers.

    Proper fix would involve modifying the entrypoint scripts to:

    1. Remove tail -f /dev/null
    2. Run Java as PID 1 (foreground, no &) so the container exits when Java dies
    3. Use dumb-init (already installed in Store/PD images) for signal handling

    There's also an unused cron-based monitor in the Server component (enabled via -m flag in start-hugegraph.sh), but it's not activated in the Docker entrypoint and only covers Server — not Store or PD.

    To reply, just mention @dosu.


    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about hugegraph with Dosu Add Dosu to your team

  2. bitflicker64 commented on May 30, 2026

    @bitflicker64
    ContributorAuthor

    Implementation Plan

    Breaking the work into independently reviewable and testable chunks. Each chunk has its own test script, can be reviewed in isolation, and builds on the previous one.


    Chunk 1 — Fix start-hugegraph.sh foreground mode

    File: hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh

    Problem: All post-branch logic (PID="$!", pid file write, trap, wait_for_startup, disown, OPEN_MONITOR) runs unconditionally after the daemon/foreground if/else block. In foreground mode the script blocks at hugegraph-server.sh until Java exits, then all those lines run with $! empty/stale, wait_for_startup immediately fails (Java already dead), and the script exits 0 losing Java's exit code entirely.

    Fix: Move all post-branch logic inside the DAEMON == "true" branch. Add exit $? to the foreground branch so Java's exit code propagates out.

    Test: hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph.sh


    Chunk 2 — Add -d flag to start-hugegraph-pd.sh

    File: hugegraph-pd/hg-pd-dist/src/assembly/static/bin/start-hugegraph-pd.sh

    Problem: No foreground mode exists — Java is always backgrounded unconditionally with exec java ... &. No -d flag. wait_for_startup exists in pd's util.sh but is never called — daemon startup has no health polling at all, callers compensate with hardcoded sleep 10.

    Fix: Add DAEMON="true" default (consistent with existing style in this file), add -d to getopts, split the exec java block into daemon/foreground branches, wire up wait_for_startup in daemon mode, add exit $? to foreground branch.

    Test: hugegraph-pd/hg-pd-dist/src/assembly/travis/test-start-hugegraph-pd.sh


    Chunk 3 — Add -d flag to start-hugegraph-store.sh

    File: hugegraph-store/hg-store-dist/src/assembly/static/bin/start-hugegraph-store.sh

    Problem: Identical to chunk 2 — no foreground mode, always backgrounds Java unconditionally.

    Fix: Identical structure to chunk 2.

    Test: hugegraph-store/hg-store-dist/src/assembly/travis/test-start-hugegraph-store.sh


    Chunk 4 — Fix pd entrypoint

    File: hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh

    Problem:

    ./bin/start-hugegraph-pd.sh -j "${JAVA_OPTS:-}"
    tail -f /dev/null   # ← zero supervision

    Fix: Pass -d false now that pd has foreground mode. Remove tail -f /dev/null. When Java exits the script exits, container exits, restart policy fires.

    ./bin/start-hugegraph-pd.sh -d false -j "${JAVA_OPTS:-}"
    # no tail needed — blocks until Java exits, exits with Java's exit code

    Chunk 5 — Fix store entrypoint

    File: hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh

    Problem: Same as chunk 4.

    Fix: Same as chunk 4 but for store.


    Chunk 6 — Fix server entrypoint

    File: hugegraph-server/hugegraph-dist/docker/docker-entrypoint.sh

    Problem:

    ./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
    # post-startup checks including wait-partition.sh for hstore...
    tail -f /dev/null   # ← zero supervision

    Why -d false alone doesn't work here: wait-partition.sh must run after the server is already up in the background. The structure must stay: start in background → post-startup checks → then supervise.

    Fix: Keep the background start and all post-startup checks unchanged. Replace only tail -f /dev/null with wait $(cat ./bin/pid). When Java exits wait returns immediately, entrypoint exits with non-zero, container exits, restart policy fires.

    ./bin/start-hugegraph.sh -j "${JAVA_OPTS:-}" -t 120
    
    # post-startup checks preserved exactly as-is
    ACTUAL_BACKEND=$(...)
    if [[ "${ACTUAL_BACKEND}" == "hstore" ]]; then
        ./bin/wait-partition.sh || log "WARN: partitions not assigned yet"
    fi
    
    # replace tail -f /dev/null:
    wait "$(cat ./bin/pid)"
    exit $?

    Chunk 7 — Add HEALTHCHECK to all four Dockerfiles

    Files: hugegraph-server/Dockerfile, hugegraph-server/Dockerfile-hstore, hugegraph-pd/Dockerfile, hugegraph-store/Dockerfile

    Problem: No HEALTHCHECK in any Dockerfile. Health checks only exist in docker-compose.yml. Plain docker run has no health reporting. depends_on: condition: service_healthy relies on compose injecting the check at runtime rather than it being baked into the image.

    Fix: Add HEALTHCHECK to each Dockerfile before ENTRYPOINT, mirroring the exact values already in docker-compose.yml:

    # hugegraph-pd/Dockerfile
    HEALTHCHECK --interval=10s --timeout=5s --retries=12 --start-period=30s \
        CMD curl -fsS http://localhost:8620/v1/health >/dev/null || exit 1
    
    # hugegraph-store/Dockerfile
    HEALTHCHECK --interval=10s --timeout=10s --retries=30 --start-period=60s \
        CMD curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1
    
    # hugegraph-server/Dockerfile + Dockerfile-hstore
    HEALTHCHECK --interval=10s --timeout=5s --retries=30 --start-period=60s \
        CMD curl -fsS http://localhost:8080/versions >/dev/null || exit 1

    Chunk 8 — Remove cron from all four Dockerfiles

    Files: All four Dockerfiles

    Problem: cron is installed but crond is never started — it has no purpose in the image now that supervision is Docker-native.

    Fix: Remove cron from the apt-get install step in all four Dockerfiles. Keep dumb-init, procps, curl, lsof.

    What stays untouched:

    • monitor-hugegraph.sh — preserved, still valid for VM deployments
    • start-monitor.sh — preserved, still valid for VM deployments
    • docker-compose.yml healthcheck definitions — already correct, no changes needed

    Chunk 9 — Fix default restserver.url scheme

    File: hugegraph-server/hugegraph-dist/src/assembly/static/conf/rest-server.properties

    Problem: Shipped default is restserver.url=127.0.0.1:8080 — no http:// scheme. wait_for_startup passes this raw value to curl. On macOS, curl fails with "Protocol not supported" causing start-hugegraph.sh to always exit 1 on local development even though the server starts correctly. On Linux, curl happens to handle scheme-less URLs by defaulting to HTTP — masking the bug.

    Every other config in the repo uses http:// explicitly: raft CI configs, the Dockerfile sed patch, cluster test templates, and Java ServerOptions default. The shipped default is the only inconsistency.

    Fix: Change line 3 of rest-server.properties from 127.0.0.1:8080 to http://127.0.0.1:8080.


    Chunk 10 — Docs

    Update relevant README or docker/ docs to explain:

    • Docker deployments use native process supervision — the entrypoint watches the Java process and exits when it dies, triggering restart: unless-stopped
    • HEALTHCHECK is now baked into the Dockerfiles so docker ps reports health without needing compose
    • start-hugegraph.sh -m true (cron-based monitor) is for VM/bare-metal deployments only — unchanged and fully functional

    Blast radius

    All existing callers of start-hugegraph.sh, start-hugegraph-pd.sh, and start-hugegraph-store.sh pass no -d flag and rely on the default DAEMON="true". Daemon mode behavior is completely unchanged. Only new callers that explicitly pass -d false (the Docker entrypoints after this PR) are affected by chunks 1-3.

  3. bitflicker64 commented on May 30, 2026

    @bitflicker64
    ContributorAuthor

    @dosu what do u think about the implementation plan

  4. dosubot commented on May 30, 2026

    @dosubot

    The plan is well-structured and technically sound. A few observations after verifying against the current code:

    Chunk 1 — Looks correct. The server's start-hugegraph.sh already has a DAEMON variable and -d flag [1], but the post-branch logic (PID="$!", trap, wait_for_startup, disown, monitor) does run unconditionally after the if/else block [2]. Moving that into the daemon branch and adding exit $? to the foreground branch is the right fix.

    Chunks 2 & 3 — Confirmed. PD and Store both unconditionally background Java with & and have no -d flag [3] [4]. Adding a daemon/foreground split mirrors the pattern already established in the server script.

    Chunk 6 — One nuance worth noting. The plan uses wait "$(cat ./bin/pid)" after the post-startup checks. This works because start-hugegraph.sh writes the PID to $BIN/pid [5]. However, since dumb-init is the ENTRYPOINT in PD/Store Dockerfiles [6] but not in the server Dockerfile, you'll want to confirm that wait on a child PID works correctly under the shell that runs the entrypoint. If the server Dockerfile doesn't use dumb-init, the backgrounded Java process is a direct child of the entrypoint bash shell, so wait $PID should work fine. Just noting it's a difference from the PD/Store containers.

    Chunk 8 — Confirmed. Both PD and Store Dockerfiles do install cron via apt-get [7] [8], and the service is started during init. Removing it is clean since supervision is now Docker-native.

    Chunk 9 — Confirmed. The shipped default is indeed 127.0.0.1:8080 with no scheme [9]. Adding http:// aligns it with every other config in the repo.

    Overall assessment: The blast radius analysis is accurate — all existing callers use the default DAEMON="true" so daemon mode is untouched. The chunking is well-sequenced (scripts first, then entrypoints that depend on them, then Dockerfiles, then docs). Each chunk is independently testable.

    One suggestion: in Chunk 6, consider adding a comment in the entrypoint explaining why the server uses the start-in-background → post-checks → wait $PID pattern instead of -d false like PD/Store, since future maintainers might otherwise "simplify" it and break wait-partition.sh.

    To reply, just mention @dosu.


    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about hugegraph with Dosu Add Dosu to your team

  5. imbajin commented on May 30, 2026

    @imbajin
    Member

    Thanks for opening this. I think this is a valid follow-up to #3025, not something already fixed by it.

    #3025 intentionally kept runtime behavior unchanged and focused on Docker build/cache hygiene. This issue is about a different layer: Docker runtime lifecycle and process supervision.

    Core Context

    The main problem is that the Java process lifecycle is currently decoupled from the container lifecycle.

    At the moment, the Docker entrypoints start the HugeGraph Java process in the background, then keep the container alive with tail -f /dev/null.

    That means when Java dies, Docker still sees the container as running, so restart: unless-stopped does not trigger.

    Current behavior
    
    ┌──────────────────────────────┐
    │ dumb-init / docker-entrypoint │
    └──────────────┬───────────────┘
                   │
                   ▼
           start-*.sh starts Java &
                   │
                   ▼
           tail -f /dev/null keeps running
                   │
                   ▼
            Java crashes or exits
                   │
                   ▼
           container still appears Up
                   │
                   ▼
           Docker restart policy is not triggered
                   │
                   ▼
           users get connection refused
    

    Expected Docker-native behavior should be closer to:

    Expected behavior
    
    ┌──────────────────────────────┐
    │ dumb-init / docker-entrypoint │
    └──────────────┬───────────────┘
                   │
                   ▼
           supervise actual Java process
                   │
                   ▼
            Java crashes or exits
                   │
                   ▼
              entrypoint exits
                   │
                   ▼
              container exits
                   │
                   ▼
           Docker restart policy restarts it
    

    Conclusion

    I agree this should be treated as a real Docker runtime lifecycle bug.

    The key points are:

    • tail -f /dev/null hides Java process failure from Docker.
    • Docker restart policies only trigger when the container exits.
    • A Compose/Docker healthcheck helps observability, but ordinary Docker does not restart a container just because it is unhealthy.
    • The legacy cron monitor is not a good Docker supervision model unless cron is actually started, and even then it is less aligned with Docker-native lifecycle handling.
    • PD and Store need a real foreground mode.
    • Server needs extra care because the hstore path has post-start checks such as wait-partition.sh; it likely needs the pattern: start in background, run post-start checks, then wait on the actual Java PID.

    Suggested Fix Direction

    A clean fix should probably be split like this:

    1. Fix startup script foreground semantics

      • Keep daemon mode as the default for existing callers.
      • Fix start-hugegraph.sh -d false so it propagates the Java exit code correctly.
      • Add foreground mode to start-hugegraph-pd.sh.
      • Add foreground mode to start-hugegraph-store.sh.
    2. Fix Docker entrypoints

      • PD / Store: run the Java process in foreground mode and remove tail -f /dev/null.
      • Server: preserve the current post-start hstore checks, then wait on the real HugeGraphServer PID instead of tailing forever.
    3. Document the lifecycle model

      • HEALTHCHECK reports health.
      • restart: unless-stopped restarts only after container exit.
      • Docker deployments should rely on container lifecycle, not a hidden background Java process plus tail.

    Validation

    The fix should include a runtime restart validation, not only a Docker build check.

    Example validation flow:

    docker build -f hugegraph-server/Dockerfile -t hugegraph/server:test .
    
    HUGEGRAPH_VERSION=test docker compose -f docker/docker-compose.yml up -d
    
    # Kill the Java process, not the container or entrypoint.
    docker exec hg-server pkill -f HugeGraphServer
    
    # The container should exit and be restarted by Docker policy.
    docker inspect hg-server \
      --format "{{.RestartCount}} {{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}"
    
    # Service should eventually recover.
    curl -fsS http://localhost:8080/versions

    Expected result after the fix:

    • The Java process death causes the entrypoint to exit.
    • The container exits instead of staying Up with a dead service.
    • RestartCount increases.
    • /versions becomes reachable again after restart.

    The same lifecycle check should be covered for Server, PD, and Store. For Server with hstore, we should also verify that the existing post-start partition wait logic is not skipped.

  6. bitflicker64 commented on May 30, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin Thanks for the validation and the runtime restart test suggestion. I'll start with the startup script changes (chunks 1–3), get the foreground mode behavior and tests in place first, then move on to the Docker entrypoint/runtime lifecycle changes in a follow-up PR.

  7. bitflicker64 commented on May 31, 2026

    @bitflicker64
    ContributorAuthor

    hey @imbajin chunk 1 PR up: #3044
    fixes foreground mode in start-hugegraph.sh, adds test script (all green on CI). chunks 2–3 next.

  8. bitflicker64 commented on Jun 4, 2026

    @bitflicker64
    ContributorAuthor

    hey @imbajin chunks 2–3 PR up: #3047
    same foreground mode fix for start-hugegraph-pd.sh and start-hugegraph-store.sh, tests wired into pd-store-ci.yml.

  9. bitflicker64 commented on Jun 5, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin chunk 4 PR up: #3051 replaces tail -f /dev/null in all three Docker entrypoints with proper Java process supervision. Also, not sure if you've been checking Discord recently , had a quick question about the Feishu access issue When you get a chance, could you take a quick look ?

  10. bitflicker64 commented on Jun 5, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin chunk 7+8 PR up: #3052 ,adds healthchecks to all four Dockerfiles (server/PD/Store/hstore) and removes unused cron from the images.

  11. bitflicker64 commented on Jun 8, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin — the core fix is complete. All merged PRs:

    #3044 — fix foreground mode exit code propagation in start-hugegraph.sh
    #3047 — same fix for start-hugegraph-pd.sh and start-hugegraph-store.sh
    #3051 — replace tail -f /dev/null with proper Java process supervision in all three Docker entrypoints
    #3052 — add HEALTHCHECK to all four Dockerfiles and remove unused cron from images

    Also have a non-blocking CI hardening PR open (#3055) to distinguish "skipped" from "passed" in the startup test suite, flagged in your reviews of #3044 and #3047.

    Starting on the documentation PR (Chunk 10) next — will cover the -d true|false flag, Docker-native supervision model, and HEALTHCHECK behavior.

    Is there anything else you'd like addressed before we close this out?

  12. bitflicker64 commented on Jun 9, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin Both docs PRs are now open — Chunk 10 complete:

    Once #3055, #3056, and the hugegraph-doc PR merge, this issue can be closed.

  13. bitflicker64 commented on Jun 11, 2026

    @bitflicker64
    ContributorAuthor

    @imbajin I think this issue can be safely closed, unless there's something I'm overlooking, of course.
    Also, is there anything else you'd like me to take care of?
    I had a couple of questions as well. Would Discord still be the best place to ask them, or would you prefer that I send an email instead?

  14. imbajin commented on Jun 12, 2026

    @imbajin
    Member

    @imbajin I think this issue can be safely closed, unless there's something I'm overlooking, of course. Also, is there anything else you'd like me to take care of? I had a couple of questions as well. Would Discord still be the best place to ask them, or would you prefer that I send an email instead?

    thanks & lgtm now

    for questions: both fine for me (discord and mail)

  15. added a commit that references this issue on Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions