Skip to content

Remote Control app-server survives Homebrew upgrade from .upgrading path, causing stale codex-code-mode-host spawn failure #44261

Open

Description

@maxoyed

Summary

On macOS, a long-lived Codex Remote Control app-server can survive a Homebrew Cask upgrade after being launched from Homebrew's transient .upgrading directory. After the upgrade completes and that directory is removed, the old app-server continues running and later fails to spawn codex-code-mode-host because it resolves the helper relative to the stale install path.

This manifests when controlling a Mac ChatGPT/Codex session remotely from the iPhone ChatGPT app.

Environment

  • macOS on Apple Silicon
  • ChatGPT Desktop: latest version at time of report
  • Bundled Codex: codex-cli 0.153.4
  • Homebrew Cask Codex current version: 0.153.4
  • Remote Control used from the iPhone ChatGPT app

Current Homebrew installation is complete:

/opt/homebrew/Caskroom/codex/0.153.4/bin/codex
/opt/homebrew/Caskroom/codex/0.153.4/bin/codex-code-mode-host

/opt/homebrew/bin/codex resolves correctly to:

/opt/homebrew/Caskroom/codex/0.153.4/bin/codex

CODEX_CLI_PATH is not set in either the shell or launchctl environment.

Error

Remote Code Mode reported:

failed to spawn code-mode host /opt/homebrew/Caskroom/codex/0.153.0/bin/codex-code-mode-host: No such file or directory (os error 2)

However, the current 0.153.4 installation contains codex-code-mode-host normally.

Root cause evidence

A stale Remote Control app-server had been running for several days:

2330  2322  Fri Sep 4 10:02:53 2026  codex -c features.code_mode_host=true app-server --listen unix://

Inspecting the executable mapped into that running process with lsof showed:

codex 2330 ... txt REG ... /opt/homebrew/Caskroom/codex/0.153.0.upgrading/bin/codex

So the Remote app-server had actually been launched while Homebrew was in the middle of upgrading Codex and the Cask directory had temporarily become:

/opt/homebrew/Caskroom/codex/0.153.0.upgrading/bin/codex

After the Homebrew upgrade completed, the transient/old directory was removed and the current installation became 0.153.4, but the old app-server process continued to run.

Later, when Code Mode was invoked through Remote Control, that stale process attempted to launch the sibling codex-code-mode-host from the old 0.153.0 installation path and failed with os error 2.

Expected behavior

A Remote Control / persistent app-server should not remain bound to a transient Homebrew .upgrading executable path after the package is replaced.

Possible expected behaviors:

  1. Detect that the executable/runtime backing the daemon has been replaced or removed and restart the app-server.
  2. Pin/copy a stable managed runtime before launching a persistent Remote Control daemon.
  3. Avoid launching a long-lived app-server from a Homebrew .upgrading path.
  4. Resolve codex-code-mode-host from a stable runtime location rather than a path that can disappear during package upgrades.

Actual behavior

The app-server survives the Homebrew upgrade and remains alive even though its original executable path no longer exists on disk. Code Mode later fails when attempting to spawn its sibling helper.

Workaround

Terminating the stale Remote app-server and its bootstrap parent fixes the issue:

kill -TERM 2330
kill -TERM 2322

After restarting/reconnecting Remote Control, the stale 0.153.0 / .upgrading references disappear and current ChatGPT Desktop app-servers use:

/Applications/ChatGPT.app/Contents/Resources/codex
/Applications/ChatGPT.app/Contents/Resources/codex-code-mode-host

The Homebrew installation also remains healthy at 0.153.4.

Related issues

This seems related to reports of Remote/app-server version skew and Code Mode host path issues, including #31835 and #31907, but this case specifically captures the Homebrew .upgrading race/lifecycle condition with direct lsof evidence.

Activity

  1. added
    bugSomething isn't working
    CLIIssues related to the Codex CLI
    app-serverIssues involving app server protocol or interfaces
    on Sep 9, 2026
  2. argszero commented on Sep 9, 2026

    @argszero

    Thanks for the thorough report — the lsof evidence (the live app-server still mapped to 0.153.0.upgrading/bin/codex) is the smoking gun. I traced the resolver on current main and can confirm the mechanism precisely.

    Root cause: InstallContext::current() is a process-wide OnceLock initialized once from current_exe() and never re-resolved.

    The code-mode host binary is resolved in codex-rs/install-context/src/lib.rs:

    • InstallContext::current() uses INSTALL_CONTEXT.get_or_init(...), which calls Self::from_exe(..., current_exe.as_deref(), ...) where current_exe = std::env::current_exe(). That snapshot is taken once, at process start, and is immutable for the whole process lifetime (OnceLock).
    • code_mode_host_program() (lib.rs) → code_mode_host_program_from_exe(current_exe) derives the sibling host path next to the process executable (or from the package layout's bin_dir).

    So when a long-lived Remote Control app-server is launched while Homebrew's cask directory is the transient 0.153.0.upgrading/bin/codex, current_exe() returns that stale path. After the upgrade completes and Homebrew removes the .upgrading directory, the running daemon still holds the old InstallContext, and code_mode_host_program() keeps pointing at the deleted 0.153.0.upgrading/bin/codex-code-mode-host.

    The failure then surfaces exactly where you saw it:

    • codex-rs/code-mode/src/remote_session/connection.rs:84 emits "failed to spawn code-mode host {host_program}: {error}" — the {host_program} is that stale path, and {error} is No such file or directory (os error 2).

    Note this is not merely a "the host binary is missing" problem — the correct current codex-code-mode-host still exists (you confirmed 0.153.4 has it normally). The daemon just resolves the wrong (stale, deleted) per-process path and never revalidates.

    Why your workaround works: killing the stale app-server (and its bootstrap parent) forces the next Remote Control launch to current_exe() into the real install, so 0.153.4/bin paths get used again.

    On your proposed options — they're all reasonable, and the most robust is #4 plus a staleness guard:

    • The InstallContext is deliberately cached for process lifetime (an OnceLock), so re-resolving current_exe() on every code_mode_host_program() call would be the targeted fix, but it also rebuilds method/package_layout each time. A cheaper, lower-risk version: resolve the host program at spawn time rather than caching it in the once-locked context, or re-check that the cached host_program still exists on disk and, if not, fall back to current_exe()'s current parent / to the bundled codex-resources copy (which is the preferred location anyway).
    • On macOS, avoiding Brew-transient paths generally also helps: a Homebrew cask upgrade renames the directory (0.153.0 → 0.153.0.upgrading → 0.153.4), so a path resolved off current_exe() at launch, then cached forever, is inherently vulnerable to the exact race you captured.

    I'm happy to put up a fix. Would you prefer the spawn-time re-resolution (survives the cache) or the "validate cached host program still exists, else re-resolve from current_exe/resources" fallback? I can adjust to whichever shape the maintainers favour.

  3. martinezhermes commented on Sep 23, 2026

    @martinezhermes

    I run a personal Hermes Agent fork on a dedicated Mac mini. My ChatGPT Pro account supplies most of its everyday inference. I also have a multi-GPU cluster for experimenting with RL-trained Qwen models, but that generates enough heat that I track it in Home Assistant. I generally save the local compute for low-latency multimodal work. The whole setup is partly an autoresearch-style experiment in recursive, self-supervised improvement, and partly just Hermes doing agent things.

    I was upgrading Codex because Hermes could not list GPT-6 models. Upgrading an autonomous agent’s own runtime is awkward for me and for themselves: it has to switch providers, repoint its access paths, and restart the processes it depends on. During this, the remote app-server kept reappearing after resets. For a moment, it looked like Hermes was protecting its setup, and it was very hilarious to watch and debug.

    Observed on the Mac mini: macOS arm64, Homebrew Codex CLI 0.156.1, ChatGPT desktop 26.917.51856.

    • An agent could chat, but the Codex harness was broken, even pwd timed out before reaching the shell: timed out negotiating with the code-mode host. This happened with any model.
    • Reinstalling the CLI and desktop app did not resolve it.
    • The Homebrew codex-code-mode-host binary hung even on --help. A byte-identical, signed copy in the same directory returned help immediately.
    • Atomically replacing the original helper with that same signed copy restored execution. The remote task then ran pwd under gpt-6-sol in 0.1 seconds.

    Expected: If the helper cannot start or negotiate, the remote app-server should identify the failing executable and recover or give an actionable error. A session that can still chat while every tool call times out is difficult to diagnose, especially when the agent is trying to upgrade its own runtime.

    I have not established whether the missing GPT-6 model entries share a cause with this tool failure. That was the reason for the upgrade; the code-mode timeout is the failure I could isolate and repair.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CLIIssues related to the Codex CLIapp-serverIssues involving app server protocol or interfacesbugSomething isn't workingremote

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions