Skip to content

amplifier update: stale editable installs left pointing at deleted cache dirs → silent provider load failures #299

Open

Description

@Joi

Symptoms

After running amplifier update, two of four configured providers (openai, gemini) silently failed to load. The only visible signal was:

Partial provider failure: 2/4 loaded. Missing: {'openai': 1, 'gemini': 1}.
Loaded: ['anthropic', 'github-copilot']. Session continuing with available providers
(self-healing NOT triggered for partial failure).

The actual exception (ModuleNotFoundError: No module named 'amplifier_module_provider_openai') is swallowed before reaching the user-visible log. Self-healing did not trigger because the failure was classified as "partial."

The same amplifier update run also produced this clone error at the top, which appears to be correlated:

✗ Bundle: joi: Failed to clone ssh://git@github.com/Joi/amplifier-bundle-joi@main:
fatal: could not open '/Users/joi/.amplifier/cache/amplifier-bundle-joi-<hash>/.git/objects/pack/tmp_pack_dnAj5l' for reading: No such file or directory
fatal: fetch-pack: invalid index-pack output

amplifier update ran git clone into a populated cache dir instead of git pull, failed mid-pack, but left the bundle in its previously healthy state.

Root cause

amplifier update re-clones modules into content-hash-suffixed cache directories when upstream changes. For openai+gemini on the affected run, the re-clone produced new hash dirs, but the editable install (uv pip install -e) was not re-run against the new paths. The old hash dirs were subsequently removed by the updater's GC step, leaving the editable install pointers dangling.

Observed state on the affected machine (uv tool install-managed amplifier venv at ~/.local/share/uv/tools/amplifier/):

Module .pth and .dist-info/direct_url.json pointed to Cache currently contained
openai cache/amplifier-module-provider-openai-e2c4b8c10222ed8c (deleted) …-f62df101b02f7b3e
gemini cache/amplifier-module-provider-gemini-06d1437d03d6b064 (deleted) …-055b52ecbb4ac482
anthropic (control, working) …-5181591dcf06d076 …-5181591dcf06d076 ✓

import amplifier_module_provider_openai → ModuleNotFoundError because the .pth pointed to a non-existent directory. Anthropic and github-copilot worked because their content hashes hadn't changed and their install pointers were still consistent.

The "tmp_pack_dnAj5l" clone error at the top of the update run is likely related: the bundle clone failure may have caused the updater to abort its post-clone editable-install consistency check, leaving modules whose hash DID change in an inconsistent state.

Surgical fix that worked

Without touching the cache directories:

AMP_PYTHON=~/.local/share/uv/tools/amplifier/bin/python
uv pip install --python "$AMP_PYTHON" -e ~/.amplifier/cache/amplifier-module-provider-openai-<current-hash>
uv pip install --python "$AMP_PYTHON" -e ~/.amplifier/cache/amplifier-module-provider-gemini-<current-hash>

uv pip install cleanly reported the swap from the dead path to the live one:

- amplifier-module-provider-openai==1.0.0 (from file:///…/openai-e2c4b8c10222ed8c)
+ amplifier-module-provider-openai==1.0.0 (from file:///…/openai-f62df101b02f7b3e)

After this, Partial provider failure disappeared and all 4 providers loaded.

The heavier alternative amplifier reset --remove cache -y && amplifier update would also have worked but re-clones everything.

Diagnostic indicators (for other users hit by this)

  1. amplifier session start prints Partial provider failure: … with {'<provider>': N, ...}
  2. <venv>/lib/python3.*/site-packages/_editable_impl_amplifier_module_provider_*.pth points at a ~/.amplifier/cache/... directory that no longer exists on disk
  3. The matching *.dist-info/direct_url.json confirms the stale URL

Suggested fixes

  1. Surface the actual exception in the partial-failure message. Today it just shows a count ({'openai': 1, 'gemini': 1}) — include the underlying ImportError / ModuleNotFoundError so users can diagnose without dropping into the venv.
  2. Treat editable-install consistency as a post-update invariant. After cloning modules to new hash dirs, verify (via direct_url.json or equivalent) that every editable install points at a path that still exists; reinstall against the current path if not. Sub-millisecond check.
  3. Prefer git pull over git clone when the cache dir already contains a valid git repo with a matching origin URL. The tmp_pack_dnAj5l error would not have happened, and the partial pipeline failure would not have left consistency gaps downstream.
  4. Don't treat "partial provider failure" as silently OK. It's the worst kind of failure for a system that routes by role — a session configured with model_role: critical-ops against openai will silently downgrade to anthropic with no signal to the calling agent. Consider non-zero exit and/or hard-fail by default with an explicit --allow-partial-providers flag if quiet downgrade is desired.

Environment

  • macOS 26.5 arm64 (M-series)
  • amplifier installed via uv tool install amplifier
  • Python 3.12 (uv-managed)
  • Bundle: a private bundle composed against microsoft/amplifier-foundation + microsoft/amplifier-bundle-routing-matrix
  • All 4 provider modules sourced from git+https://github.com/microsoft/amplifier-module-provider-*@main

Happy to attach pip list, the affected .pth contents, or the cache directory listing if useful.

Activity

  1. kenotron-ms commented on May 27, 2026

    @kenotron-ms
    Contributor

    Action Plan: Verification Infrastructure + Fix

    What We're Proposing

    Before landing the ~10 line fix to update_module(), we're building verification infrastructure that proves the install/update pipeline works across all scenarios. The fix is straightforward -- add ModuleActivator._install_dependencies() after resolver.resolve() in module_cache.py, mirroring what update_bundle() already does correctly. The harder problem is: how do we prove it works and gate future changes?

    The approach: extend amplifier-tester's existing validator with a new install-update check type and create 3 DTU profiles that exercise the full install/update lifecycle.

    Scenarios Covered

    Normal User (install -> use -> update -> use):

    • U1: Fresh install from git
    • U2: First-run provider setup (check_first_run triggers ensure_provider_installed)
    • U3: Update with no hash changes (nothing should break)
    • U4: Update with hash changes -- THE BUG -- .pth files must point at new dirs
    • U5: Partial hash change (2/4 providers changed, 2/4 didn't -- all must remain importable)
    • U6: Umbrella self-update (amplifier-app-cli itself updates, venv rebuilds)
    • U7: Reset + reinstall (amplifier reset --remove cache -> clean slate)

    Developer (source overrides + update):

    • D1: Local path override via amplifier source add provider-X ~/dev/X
    • D2: Local override survives module update (override untouched, other modules updated correctly)
    • D3: Local override survives umbrella update (transient breakage by design, recovery on next session)
    • D4: Git fork override via amplifier source add provider-X git+https://fork@branch
    • D5: Git fork override with new commits -- SAME BUG AS U4 -- same fix closes it
    • D6: Ecosystem dev via amplifier-dev (DTU URL rewrite + amplifier update --force inside container)

    Regression checks (scenarios that work today, must not break):

    • U1, U2, U3, U6, U7, D1, D2, D3, D4

    Newly verified (works today but never tested):

    • D6: Post-update session viability in amplifier-dev DTU (currently only checks amplifier --version, never starts a session)

    Assertion Layer

    Extending the amplifier-tester validator with baseline checks that run after every lifecycle phase:

    • PTH_INTEGRITY: Every .pth in site-packages pointing at ~/.amplifier/cache/ references a directory that exists
    • IMPORT_SMOKE: python -c "import amplifier_module_provider_<X>" succeeds for every configured provider
    • STATE_CONSISTENT: Every entry in install-state.json has a path key that exists on disk
    • SESSION_VIABLE: amplifier run "Say exactly: install-verify-ok" completes successfully

    Plus phase-specific assertions for post-install, post-update, and post-override phases.

    DTU Profiles

    Three profiles, one per persona track:

    1. amplifier-install-update-lifecycle.yaml (Normal User) -- Fresh install, first-run setup, then simulated hash changes via cache dir deletion to reproduce the bug deterministically. Covers U1-U7.

    2. amplifier-source-override-lifecycle.yaml (Module Developer) -- Install + configure, then exercise both local path and git fork overrides through update cycles. Covers D1-D5.

    3. amplifier-dev-ecosystem-lifecycle.yaml (Ecosystem Developer) -- Install via amplifier-dev bundle with Gitea URL rewrites, then update and verify session viability (the gap in the current self-validation profile). Covers D6.

    Proposed Sequence

    1. Land verification infrastructure first (profiles + validator extension) -- U4, U5, D5 will FAIL, proving the verification catches the bug
    2. Land the ~10 line fix in amplifier-app-cli/utils/module_cache.py -- U4, U5, D5 flip green
    3. Wire profiles into CI as a gate for install-pipeline PRs

    The Fix (for reference)

    After resolver.resolve(uri) returns the new path in update_module() (~line 514):

    if (result.active_path / "pyproject.toml").exists():
        ModuleActivator._install_dependencies(result.active_path)

    Mirrors update_bundle() at amplifier-foundation/updates/__init__.py:260-268. Closes the gap for all module types in one change.


    Joi -- does this cover what you'd expect? Interested in your read on whether the three persona tracks capture all the scenarios you care about.

  2. Joi commented on Jul 5, 2026

    @Joi
    Author

    Still present on amplifier-app-cli main @ ba1957266b8dc72c16b194eb46727caf5b7cbc0d. amplifier update routes modules through execute_updates → execute_selective_module_update → update_module (amplifier_app_cli/utils/module_cache.py:468-521), which calls clear_module_cache() at :502 and SimpleSourceResolver.resolve() at :514 and returns — the editable install is never refreshed, so the venv .pth keeps pointing at the deleted hash dir. The bundle path already does this correctly: update_bundle() re-runs ModuleActivator._install_dependencies() on every updated module (amplifier-foundation amplifier_foundation/updates/__init__.py:261-268 @ 9ec8dee9). The fix described in the 2026-05-27 action plan has not landed. I closed #298 as a duplicate of this issue.

  3. Joi commented on Jul 29, 2026

    @Joi
    Author

    Update. Two PRs merged on amplifier-app-cli yesterday (2026-07-28) change the picture, but do not close this.

    microsoft/amplifier-app-cli#238 (ad69c69) aligned install-time provider source resolution with runtime resolution. microsoft/amplifier-app-cli#239 (145ba2ef) made the first-run repair path repair all configured provider modules rather than only the active one. #239's problem statement is exactly the symptom reported here — "After amplifier update or amplifier reset, only one provider module ends up installed. Every other configured provider silently fails to mount." So the user-visible failure should now self-heal on the next boot via check_first_run().

    The root cause in this issue is unchanged. On current main, update_module() in amplifier_app_cli/utils/module_cache.py still ends:

        resolver = SimpleSourceResolver(cache_dir=cache_dir)
        result = await resolver.resolve(uri)
    
        logger.debug(f"Updated {module_id}@{ref} to {result.active_path}")
    
        return result.active_path

    There is no _install_dependencies() call or other editable-install refresh after resolve, and no reference to ModuleActivator anywhere in that file. So amplifier update still leaves the venv .pth pointing at the deleted hash dir — we now depend on a repair pass in the next session rather than the update itself leaving a consistent state. update_bundle() in amplifier-foundation still handles this correctly inline.

    Suggest keeping this open scoped to the update-path fix from the 2026-05-27 action plan (the ~10 line _install_dependencies() call after resolve), since the new repair path is a safety net rather than the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions