Skip to content

Test the budget poller's artifact trust filter: only default-branch artifacts may be believed (blocks the retry-cron consumer) #2148

Closed

Description

@don-petry

Story

Follow-up to #2029 (PR 2031, the dry-run budget poller). bp_download_latest_log in scripts/lib/budget-poller.sh decides which uploaded budget-poller-log artifact the poller, and soon the retry cron (#2139), will believe. It has no test at all (tests/dev-lead/unit/test_budget_poller.bats never calls it and never mocks gh api). That function is a trust boundary: if it ever selects an artifact produced by a pull-request branch run, a PR author could feed false usage numbers to the fleet. Review threads on PR 2031 asked for these tests twice; they were not delivered, so they are tracked here.

What the function must keep doing (read the merged code and confirm; do not trust this summary)

  • Select only artifacts whose workflow_run.head_branch equals the repo's default branch, not expired, newest first, across all pages (--paginate), because other branches' runs can fill page one and hide an older trusted log.
  • Allow-list the default-branch name's characters, falling back to main on anything else.
  • Return 2 when the listing call fails mid-pagination; return 1 when no artifact is usable; fall through to an older artifact when a newer one is unreadable or not valid JSON.

Scope

Add tests only, in tests/dev-lead/unit/test_budget_poller.bats or a new file next to it, calling bp_download_latest_log directly with gh and unzip replaced by a PATH shim. No live network. Do not change budget-poller.sh unless a test exposes a real bug; if one does, fix it minimally and say so.

Acceptance criteria

  1. A newer artifact from a non-default branch (for example a PR branch) is ignored in favour of an older default-branch one. This is the security test and must fail if the head_branch filter is removed (prove it by temporarily removing the filter locally and showing the test fails; say so in the PR).
  2. A single page with one matching artifact is selected.
  3. The newest trusted artifact is on page 2 while page 1 holds only other branches' artifacts: it is still found.
  4. A listing failure on a later page returns 2 and does not abort the caller.
  5. No usable artifact returns 1; an expired artifact is skipped.
  6. The newest artifact is unreadable or not valid JSON: the next older valid one is used.
  7. A default-branch name with unsafe characters falls back to main and is never interpolated into the jq filter raw.
  8. Shellcheck clean; existing tests pass unchanged; real counts quoted from the pushed head.

Blocks

#2139 (the retry cron that will act on this artifact) must not start until this has merged.

Related: #2029, PR 2031, #2139.

Activity

  1. github-actions commented on Oct 9, 2026

    @github-actions
    Contributor

    Thanks for filing this follow-up and for spelling out the trust-boundary reasoning so clearly. Since the artifact filter guards against PR-branch runs feeding false usage numbers, I'd like to confirm scope: should the tests also cover the case where the repo's default branch is renamed mid-run (for example, the API reports a different default branch than the artifacts' head_branch), or are the seven listed acceptance criteria the complete set?

  2. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #2148 — Test the budget poller's artifact trust filter: only default-branch artifacts may be believed

    Scope

    Add bats tests that call bp_download_latest_log directly with gh and unzip replaced by a PATH shim (no network). The gh shim evaluates the function's real --jq filter against per-page fixture JSON, so removing the head_branch filter is observable. No change to scripts/lib/budget-poller.sh unless a test exposes a real bug.

    Implementation Checklist

    • tests/dev-lead/unit/test_budget_poller.bats: add a gh/unzip PATH-shim helper and a new bp_download_latest_log section (file is already in the lint.yml bats list)

    Tests to Write

    • AC1: newer PR-branch artifact ignored in favour of older default-branch one (and proven to fail with the filter removed)
    • AC2: single page, one matching artifact selected
    • AC3: newest trusted artifact on page 2, page 1 only other branches → found
    • AC4: listing fails on page 2 → returns 2, dest untouched, set -e caller continues
    • AC5: no usable artifact → 1; expired artifact skipped
    • AC6: newest unreadable / not valid JSON → next older valid one used
    • AC7: unsafe default-branch name → falls back to main, never interpolated raw into the jq filter

    Test Command

    bats tests/dev-lead/unit/test_budget_poller.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    Plan Execution (intended)

    • Added a bp_download_latest_log section to tests/dev-lead/unit/test_budget_poller.bats (already in the lint.yml bats list). gh/unzip are PATH shims; the gh shim runs the function's own --jq filter over per-page fixture listings and serves page 2+ only under --paginate.
    • scripts/lib/budget-poller.sh is unchanged — no test exposed a bug.

    Mutation check (local workspace, provisional)

    Each mutation was applied to budget-poller.sh locally, the new tests run, and the file restored:

    Mutation New tests that fail
    remove head_branch filter AC1 security test + AC2 (non-main default), AC3, AC5, both AC7
    drop --paginate AC2, AC3, AC4
    remove branch allow-list both AC7
    drop .expired == false both AC5
    || return 2 → || true AC4
    drop jq -e validity check AC6
    stop after first unreadable artifact AC6

    Files touched (this workspace, not yet pushed)

    • tests/dev-lead/unit/test_budget_poller.bats: tests only

    Notes

    Real pass counts will be quoted from the pushed head in the completion record.

  4. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #2155

    The implementation for issue #2148 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  5. added a commit that references this issue on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requestssecuritySecurity-related PRs and issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions