[BUG] apm pack silently drops description/version for remote packages when metadata fetch fails (exit 0, no warning); --check-clean then passes

[vlsi]

Describe the bug

When the remote metadata fetch fails for any reason other than an HTTP 404 on an internal repository, apm pack drops description and version from every remote git-subdir entry, prints [*] Built marketplace.json, writes nothing to stderr, and exits 0. A network outage, a proxy that refuses connections, an expired token, or a rate-limited runner all produce a truncated marketplace.json that looks like a successful build. Consumers then reject the affected plugins, because description is required once emitted.

This is the general case behind #1847, which fixed one specific trigger (raw.githubusercontent.com returning 404 for INTERNAL repos) by adding a REST fallback. The reporting gap it exposed is still open: every other failure mode stays silent.

Two code sites produce the silence:

• marketplace/builder.py::_prefetch_metadata collects the thread-pool results under except Exception: pass, so a per-package fetch failure leaves no trace at default verbosity. The same method returns early under --offline without recording that remote packages were skipped.
• marketplace/output_mappers.py::_apply_field_with_precedence omits the field when neither the entry value nor the fetched value is present, and emits no diagnostic. The surrounding code does emit BuildDiagnostic warnings for smaller problems, such as a marketplace name that is not kebab-case.

The diagnosis exists but is unreachable in practice: Could not fetch remote metadata for <name> is logged at DEBUG. Running apm pack -v shows nothing; only APM_LOG_LEVEL=DEBUG surfaces it.

The second consequence is worse than the first. --check-clean regenerates the index and diffs it against the file on disk, so when the network is down on both sides, a truncated marketplace.json passes the release gate:

marketplace.json on disk	Network	apm pack --check-clean --dry-run
truncated	blocked	exit 0 — gate passes
truncated	available	exit 4 — drift detected

A CI job that runs the gate on a runner with a degraded network therefore certifies a broken index as clean.

To Reproduce

1. Create a project with an apm.yml whose marketplace declares one remote git-subdir package. The reference below is a public repository pinned to a full SHA, so no token and no ref resolution are needed:

   name: repro
   version: 1.0.0
   description: Minimal marketplace with one remote git-subdir package
   
   targets:
     - claude
   
   marketplace:
     owner:
       name: repro-owner
     outputs:
       claude: {}
     versioning:
       strategy: per_package
     packages:
       - name: adapt-nifi-flows-to-2-x
         source: Netcracker/qubership-nifi
         subdir: agent-packages/adapt-nifi-flows-to-2-x
         ref: 09c4bf708d787eba45046904ab5b40b9ac597c6b

2. Run apm pack with the network available. The entry gets "description": "Apply NiFi 1.x to 2.x upgrade advisor recommendations to exported flow JSON files." and "version": "1.2.0", both inherited from the upstream package's own apm.yml.

3. In a fresh directory with the same apm.yml, run apm pack with outbound HTTPS pointed at a closed port. Note that --offline is not passed:

   env https_proxy=http://127.0.0.1:9 HTTPS_PROXY=http://127.0.0.1:9 ALL_PROXY=http://127.0.0.1:9 apm pack

4. Observe the exit code and the output:

   [*] Built marketplace.json [claude] (1 package(s)) -> .../.claude-plugin/marketplace.json
   exit=0
   stderr: 0 bytes
   

   The written entry has lost both fields:

   {
     "name": "adapt-nifi-flows-to-2-x",
     "source": {
       "source": "git-subdir",
       "url": "Netcracker/qubership-nifi",
       "path": "agent-packages/adapt-nifi-flows-to-2-x",
       "ref": "09c4bf708d787eba45046904ab5b40b9ac597c6b",
       "sha": "09c4bf708d787eba45046904ab5b40b9ac597c6b",
       "tag_pattern": "v{version}"
     }
   }

5. Confirm the release gate accepts it. Against the truncated file from step 4:

   env https_proxy=http://127.0.0.1:9 HTTPS_PROXY=http://127.0.0.1:9 ALL_PROXY=http://127.0.0.1:9 \
     apm pack --check-clean --dry-run; echo "exit=$?"   # exit=0
   apm pack --check-clean --dry-run; echo "exit=$?"      # exit=4

6. Confirm the information is DEBUG-only. apm pack -v mentions metadata zero times; APM_LOG_LEVEL=DEBUG apm pack prints DEBUG apm_cli.marketplace.builder Could not fetch remote metadata for adapt-nifi-flows-to-2-x.

Expected behavior

A failed metadata fetch should be visible at default verbosity, and should be able to fail the build.

1. Report it. Every swallowed fetch failure, and every remote package skipped because of --offline, should surface as a warning-level BuildDiagnostic naming the package and the cause. The build summary should not read [*] Built marketplace.json (1 package(s)) when that package lost half its metadata.
2. Let CI reject it. A flag along the lines of --strict-metadata should promote those warnings to a hard failure. Exit codes 0 through 4 are taken by success, build error, schema error, --check-versions, and --check-clean, so this needs a new one.
3. Keep --check-clean honest. The gate currently compares two artifacts that can be wrong in the same way. Failing the run when the regeneration side degraded would close that hole regardless of what the file on disk contains.

Point 2 is what we need most: a marketplace publisher wants one command that either produces a complete index or fails.

Environment (please complete the following information):

• OS: macOS 15 (Darwin 25.5.0, arm64)
• Python Version: 3.12
• APM Version: 0.28.0 (installed from PyPI via uvx --from apm-cli==0.28.0); also reproduced on the 0.26.0 native binary and observed in production on 0.24.1
• VSCode Version (if relevant): N/A

Logs

Default verbosity, network blocked, no --offline:

[!] No 'license:' field in apm.yml; the SBOM will record NOASSERTION for this package. ...
[*] Built marketplace.json [claude] (1 package(s)) -> .../.claude-plugin/marketplace.json
[i] Marketplace artifacts ready:

Nothing on stderr. The same run under APM_LOG_LEVEL=DEBUG:

DEBUG apm_cli.marketplace.builder Resolved GitHub token for metadata fetch (source=gh-auth-token)
DEBUG apm_cli.marketplace.builder Could not fetch remote metadata for adapt-nifi-flows-to-2-x
  File ".../apm_cli/marketplace/builder.py", line 1026, in _fetch_remote_metadata

Additional context

This has hit us twice in production on Netcracker/qubership-ai-packages, a marketplace with 27 packages of which 10 are remote git-subdir entries. Both times a contributor ran apm pack without working network access, committed the result, and all 10 remote packages lost description and version at once. CI ran apm pack --check-versions --check-clean --dry-run on both commits and passed them. The breakage surfaced only when consumers started rejecting the packages, and was repaired by restoring the fields by hand.

Related: #1847 (the 404-on-INTERNAL-repos trigger, fixed) and #1725 (local virtual-subdirectory packages not enriched).

[github-actions]

Triage decision

accept

Proposed labels

theme/governance
area/marketplace
area/docs-site
type/bug
status/accepted
priority/high

Milestone

null

Suggested next action

Fix _prefetch_metadata in marketplace/builder.py to surface per-package fetch failures as default-verbosity BuildDiagnostic warnings, add a --strict-metadata flag that promotes those warnings to hard failures, and ensure --check-clean fails when the regeneration side experienced any skipped or failed fetches.

Suggested issue comment

Thank you for the exceptional bug report -- the reproduction steps, the `--check-clean` gate table, and the two documented production incidents on a 27-package marketplace make this one of the clearest failure-mode analyses we have received.

The three failure modes you identified are all real:

1. `_prefetch_metadata` swallows per-package fetch exceptions under `except Exception: pass`, so a degraded network produces a truncated index with exit 0 and no user-visible signal.
2. `_apply_field_with_precedence` silently omits `description` and `version` when neither local nor fetched value is present, and emits no `BuildDiagnostic` warning.
3. `--check-clean` compares two runs that can degrade in the same way, so a network outage on both sides causes the gate to certify a broken index as clean.

The fix plan you proposed is accepted:

- Raise swallowed fetch failures and `--offline` remote-skips to `warning`-level `BuildDiagnostic` at default verbosity.
- Add `--strict-metadata` (or similar) to promote those warnings to a hard failure with a new exit code.
- Fix `--check-clean` to fail when the regeneration side experienced any skipped or failed fetches, regardless of what is on disk.

The CLI reference docs for `apm pack` will need to document the new flag and exit code in the same PR.

This is accepted at high priority. The `area/marketplace` label is applied. If you would like to contribute the fix, the relevant files are `src/apm_cli/marketplace/builder.py` and `src/apm_cli/marketplace/output_mappers.py`.

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

A marketplace publisher runs apm pack in CI and gets exit 0, [*] Built marketplace.json, zero stderr output -- and a truncated index that loses description and version for every remote git-subdir package whose metadata could not be fetched. The APM "failure mode is the product" principle is maximally violated: no signal at default verbosity, and the release gate (--check-clean) actively certifies the broken output as clean when the network is down on both sides. The fix correctly targets all three failure points: surface the failure, let CI reject it, and close the gate bypass.

Supply Chain Security Expert -- Risk-Surface Reviewer

This is a governance issue (area/marketplace, theme/governance). The "fail closed" principle from the APM security model is violated: when metadata fetch cannot be performed (network degraded), the code defaults to silent success rather than warning or refusing. The --check-clean gate bypass makes this worse: a CI job can certify a broken release artifact as clean. An attacker who can degrade a runner's network access during a release build can cause a marketplace publisher to silently ship an incomplete index. The fix (warn by default, fail with --strict-metadata, fix --check-clean) closes all three vectors. theme/governance is required because this is a release-gate integrity issue.

OSS Growth Hacker -- Contributor-Tone Reviewer

Author association is NONE, but the report is one of the most detailed in recent issues: production impact, two documented incidents, a well-structured three-point fix plan, and exact file/line citations. The reply should be warm, acknowledge the production impact explicitly, and validate all three fix points. Naming the relevant files is a concrete invitation to contribute.

Python Architect -- Architecture Reviewer

Not activated -- the fix is bounded to marketplace/builder.py and marketplace/output_mappers.py. The BuildDiagnostic pattern already exists in the codebase; adding warnings for fetch failures follows the established pattern. The --strict-metadata flag and new exit code require a minor addition to the CLI argument parser but no architectural change. The --check-clean fix is a bounded state-tracking addition in the builder.

Doc Writer -- Documentation Reviewer

Activated. Adding --strict-metadata and a new exit code to apm pack requires CLI reference docs update. The apm pack page will need a new flags section and the exit-code table updated. area/docs-site is added as a secondary area label to ensure the implementing PR covers the documentation. The fix language aligns with the user vocabulary in the marketplace guide.

APM CEO -- Triage Arbiter

Two documented production incidents on a real marketplace with 27 packages and 10 remote git-subdir entries. The --check-clean gate bypass under degraded network is the most serious part: it means a CI system can certify a broken release artifact as clean. This is high-priority governance: APM's marketplace build pipeline must fail loudly when it cannot guarantee index completeness. The fix plan is concrete and the author has already identified the exact code sites. Accept at high priority with theme/governance.

{
  "decision": "accept",
  "decision_detail": "",
  "theme": "theme/governance",
  "areas": ["area/marketplace", "area/docs-site"],
  "type": "type/bug",
  "status": "status/accepted",
  "priority": "priority/high",
  "preserved_labels": [],
  "milestone": null,
  "next_action": "Fix _prefetch_metadata to warn at default verbosity on fetch failure, add --strict-metadata flag to promote warnings to hard failure, and fix --check-clean to fail when regeneration side experienced degraded fetches.",
  "comment_markdown": "Thank you for the exceptional bug report -- the reproduction steps, the `--check-clean` gate table, and the two documented production incidents on a 27-package marketplace make this one of the clearest failure-mode analyses we have received.\n\nThe three failure modes you identified are all real:\n\n1. `_prefetch_metadata` swallows per-package fetch exceptions under `except Exception: pass`, so a degraded network produces a truncated index with exit 0 and no user-visible signal.\n2. `_apply_field_with_precedence` silently omits `description` and `version` when neither local nor fetched value is present, and emits no `BuildDiagnostic` warning.\n3. `--check-clean` compares two runs that can degrade in the same way, so a network outage on both sides causes the gate to certify a broken index as clean.\n\nThe fix plan you proposed is accepted:\n\n- Raise swallowed fetch failures and `--offline` remote-skips to `warning`-level `BuildDiagnostic` at default verbosity.\n- Add `--strict-metadata` (or similar) to promote those warnings to a hard failure with a new exit code.\n- Fix `--check-clean` to fail when the regeneration side experienced any skipped or failed fetches, regardless of what is on disk.\n\nThe CLI reference docs for `apm pack` will need to document the new flag and exit code in the same PR.\n\nThis is accepted at high priority. The `area/marketplace` label is applied. If you would like to contribute the fix, the relevant files are `src/apm_cli/marketplace/builder.py` and `src/apm_cli/marketplace/output_mappers.py`."
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel · 197.1 AIC · ⌖ 9 AIC · ⊞ 9.2K · ◷