Skip to content

[foundation] Distribution/import-name mismatch rebuilds editable bundles on every process and can fill disk #326

Closed
microsoft/amplifier-foundation
#275

Description

Summary

Amplifier Foundation deterministically reinstalls some already-installed editable bundle packages on every fresh amplifier process because it assumes a Python distribution name maps to its import name via name.replace("-", "_").

For amplifier-bundle-evaluation:

  • Distribution name: amplifier-bundle-evaluation
  • Actual import package: amplifier_evaluation
  • Guessed import checked by Foundation: amplifier_bundle_evaluation

The guessed import can never resolve. Foundation therefore invalidates a valid, fingerprint-matching install-state entry and invokes uv pip install -e <bundle checkout> again on every process.

This is not a UV cache-locking failure. UV documents that local directories passed explicitly on the command line are always rebuilt/reinstalled. Amplifier is repeatedly and incorrectly asking UV to do that work.

Impact observed

On one WSL workstation during a high-process-count evaluation campaign:

  • 1,287 redundant editable builds accumulated.
  • ~/.cache/uv reached 118.5 GiB.
  • Each evaluation rebuild allocated approximately 92–97 MiB:
    • 43,887,370-byte wheel
    • approximately 52 MiB unpacked archive
  • The WSL root filesystem fell from 158 GiB free to 8 GiB free and reached 99% usage in five days.
  • uv cache prune subsequently removed 117.4 GiB across 258,938 unreachable files.
  • Resource-pressure symptoms included process/thread creation failures; one benchmark trial also failed during a redundant editable install, though the retained logs do not establish the cause of UV exit status 101.

Concurrency is not required: isolated one- and two-generation hours show sequential fresh Amplifier processes are sufficient. Concurrency only multiplies the rate.

The evaluation bundle made the incident unusually severe because its wheel force-includes a roughly 52 MiB benchmark corpus. The same faulty check appears to affect other bundles:

  • amplifier-bundle-context-intelligence distributes context_intelligence.
  • amplifier-bundle-attractor declares packages=[].
  • amplifier-bundle-dot-graph declares packages=[].

For package-less projects, no import can ever satisfy the current check.

Environment

  • Amplifier Foundation: 1.0.0
  • Foundation commit: 37a257ad0437b72ffc14af9d854cc0652a3ef2a3
  • Evaluation bundle: 0.2.0
  • Evaluation commit: ec9f4ca3a23e74a519fced2fe0e2a9ec39e5764d
  • Amplifier installed as a UV tool
  • Ubuntu under WSL

Root cause

Foundation derives an import name from [project].name and tests it with find_spec():

The evaluation distribution/import mismatch is declared here:

https://github.com/microsoft/amplifier-bundle-evaluation/blob/ec9f4ca3a23e74a519fced2fe0e2a9ec39e5764d/pyproject.toml

A read-only runtime check against the installed tool environment produced:

find_spec("amplifier_bundle_evaluation") -> None
find_spec("amplifier_evaluation")        -> valid ModuleSpec
distribution("amplifier-bundle-evaluation") -> installed version 0.2.0
packages_distributions()["amplifier_evaluation"] -> ["amplifier-bundle-evaluation"]

The install-state file already contained the evaluation checkout with a matching fingerprint. The stale-state cross-check repeated the same bad guessed-import lookup, invalidated the entry, ran uv pip install -e, then recorded the same package again.

Exact steady-state cycle:

fresh amplifier process
  -> prepare composed bundle roots
  -> guess amplifier_bundle_evaluation
  -> find_spec returns None
  -> find matching install-state fingerprint
  -> cross-check same nonexistent import
  -> invalidate valid state
  -> uv pip install -e <evaluation checkout>
  -> mark installed again

Why the bundle was present in unrelated sessions

-B anchors selects the primary bundle but does not isolate globally configured app bundles. App bundles are intentionally composed into every session. That is expected behavior, but it means one faulty app-bundle package check affects all Amplifier sessions.

There is also a design-level amplification: composing a small behavior fragment causes Foundation to install the repository's root Python project. In the evaluation repository, that project is an independent harness with a large embedded benchmark corpus, not something the behavior itself needs at runtime.

Secondary cross-process install-state bug

There is an independent lost-update race in InstallStateManager:

  • Each process loads the entire JSON snapshot once.
  • mark_installed() updates only that process's memory.
  • save() atomically replaces the whole file without a cross-process lock, reread, or merge.

Relevant code:

https://github.com/microsoft/amplifier-foundation/blob/37a257ad0437b72ffc14af9d854cc0652a3ef2a3/amplifier_foundation/modules/install_state.py#L34-L42

https://github.com/microsoft/amplifier-foundation/blob/37a257ad0437b72ffc14af9d854cc0652a3ef2a3/amplifier_foundation/modules/install_state.py#L172-L212

Two processes can therefore load state S, independently add X and Y, then save S+X followed by S+Y, losing X. This can cause additional redundant installs. It is secondary to this incident: evaluation reinstalls even when its matching state entry survives.

Suggested fixes

  1. Determine whether a distribution is installed with importlib.metadata.distribution(project_name) rather than a guessed import name.
  2. If import health must be checked, allow bundle/package metadata to declare the import name explicitly; do not infer it from the distribution name.
  3. Skip root editable installation for projects that explicitly declare packages=[].
  4. Put a cross-process lock around check -> install -> mark, then recheck after acquiring the lock.
  5. Lock, reread, and merge install state before saving, or replace the shared JSON with SQLite.
  6. Persist a successful install while holding the lock instead of waiting until the end of complete bundle preparation.
  7. Avoid installing a composed behavior repository's root project unless that behavior explicitly declares it as required.
  8. Add regression coverage for:
    • distribution/import-name mismatch
    • package-less bundle roots
    • repeated sequential process startup
    • concurrent install-state writers

Workaround

Until fixed:

  • Avoid placing the evaluation bundle in global bundle.app configuration for workloads that do not need it.
  • Isolate benchmarks with a local settings layer whose bundle.app list is empty.
  • Periodically run uv cache prune only after Amplifier/UV activity is quiescent.

Related but distinct from #299: that issue concerns stale editable pointers after cache-path rotation; this issue concerns valid installed packages being falsely classified as absent and rebuilt on every process.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions