Repository navigation
chore: fail pull requests that grow the usage CLI or a derived CLI by more than 1% #1492
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| name: size | ||
|
|
||
| # Stripped release sizes of the `usage` CLI and a mise-sized derived CLI, against the commit | ||
| # this pull request would merge onto. Fails when a gated binary grows by more than the | ||
| # threshold in `tasks/size.sh`, which says what is measured and why. | ||
| # | ||
| # A hosted runner, unlike perf-pr: size does not depend on the machine, so there is nothing | ||
| # for a dedicated one to hold still. The checkout is GitHub's merge of the pull request into | ||
| # its base, so the merge base is the base branch's tip and the difference is this change alone. | ||
| # | ||
| # Never add `continue-on-error`, raise the threshold, or drop a binary to get a pull request | ||
| # through — see "Performance Gate" in AGENTS.md. | ||
|
|
||
| on: | ||
| pull_request: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| CARGO_TERM_COLOR: always | ||
| CARGO_INCREMENTAL: "0" | ||
|
|
||
| jobs: | ||
| size: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 45 | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| # `tasks/size.sh` builds the merge base from a worktree. | ||
| fetch-depth: 0 | ||
| persist-credentials: false | ||
| - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 # zizmor: ignore[cache-poisoning] | ||
| with: | ||
| shared-key: size | ||
| # The report reaches the job summary whether or not the gate passed, and the exit | ||
| # status is raised afterwards, so a failure always arrives with the table explaining it. | ||
| - name: Compare binary sizes against the base | ||
| env: | ||
| # From the pull request payload: perf-pr found the generic `github.base_ref` | ||
| # naming main for a reopened stacked pull request. | ||
| BASE_REF: origin/${{ github.event.pull_request.base.ref }} | ||
| run: | | ||
| set +e | ||
| ./tasks/size.sh "$BASE_REF" /tmp/size-report.md | ||
| status=$? | ||
| set -e | ||
| if [ -s /tmp/size-report.md ]; then | ||
| tee -a "$GITHUB_STEP_SUMMARY" </tmp/size-report.md | ||
| else | ||
| echo "The size comparison never produced a report." | tee -a "$GITHUB_STEP_SUMMARY" | ||
| fi | ||
| exit "$status" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| #!/usr/bin/env bash | ||
| # Binary size: this checkout against the commit it branched from, stripped. | ||
| # | ||
| # Size is a property of the build, not of the machine, so unlike instruction counts it needs | ||
| # no dedicated runner and no noise margin to be trusted: two builds of the same source with | ||
| # the same toolchain come out the same size. The 1% gate is for what an adopter pays, not | ||
| # for noise. | ||
| # | ||
| # Never loosen GATE_PCT, drop a binary, or mark one ungated to get a pull request through — | ||
| # see "Performance Gate" in AGENTS.md, which applies here the same way. A change that | ||
| # genuinely costs bytes says so with these numbers, and the maintainer decides. | ||
| # | ||
| # Usage: tasks/size.sh [BASE_REF] [OUT] | ||
| # BASE_REF what to branch-point against (default: origin/main) | ||
| # OUT where to write the markdown report (default: stdout) | ||
| # Exits 1 when a gated binary grew by more than GATE_PCT. | ||
| set -euo pipefail | ||
|
|
||
| GATE_PCT=1 | ||
| base_ref=${1:-origin/main} | ||
| out=${2:-/dev/stdout} | ||
|
|
||
| base=$(git merge-base "$base_ref" HEAD) | ||
| tmp=$(mktemp -d) | ||
| trap 'git worktree remove --force "$tmp/src" 2>/dev/null || true; rm -rf "$tmp"' EXIT | ||
|
|
||
| # One target directory for both builds, so crates.io dependencies compile once. Workspace | ||
| # crates still build twice: cargo keys them by path, and the base lives in a worktree. | ||
| # Absolute, because the base builds from inside the worktree: a relative path would put its | ||
| # binaries under the worktree while `build` copies them from here. | ||
| CARGO_TARGET_DIR=${CARGO_TARGET_DIR:-target/size} | ||
| case $CARGO_TARGET_DIR in /*) ;; *) CARGO_TARGET_DIR=$PWD/$CARGO_TARGET_DIR ;; esac | ||
| export CARGO_TARGET_DIR | ||
|
|
||
| # `parse-usage` is mise's whole CLI through the derive, which is the size an adopter's binary | ||
| # pays for usage. `parse-none` is the same program without the parse, subtracted so the gate | ||
| # measures usage's share rather than std's. `parse-clap` is context and never gated. | ||
| BINS=(parse-none parse-usage parse-clap usage) | ||
|
|
||
| build() { | ||
| local src=$1 dest=$2 | ||
| # Both checkouts link to the same paths. Removing them first means a build that failed to | ||
| # relink stops `cp` here, rather than handing one checkout's binary to the other. | ||
| for bin in "${BINS[@]}"; do rm -f "$CARGO_TARGET_DIR/release/$bin"; done | ||
| (cd "$src" && cargo build --release --locked -q -p gate -p usage-cli \ | ||
| --bin parse-none --bin parse-usage --bin parse-clap --bin usage) | ||
| mkdir -p "$dest" | ||
| for bin in "${BINS[@]}"; do | ||
| cp "$CARGO_TARGET_DIR/release/$bin" "$dest/$bin" | ||
| strip "$dest/$bin" | ||
| done | ||
| } | ||
|
|
||
| bytes() { wc -c <"$1" | tr -d ' '; } | ||
|
|
||
| git worktree add -q --detach "$tmp/src" "$base" | ||
| build "$tmp/src" "$tmp/base" | ||
| build . "$tmp/head" | ||
|
jdx marked this conversation as resolved.
|
||
|
|
||
| # The shadow is a fixture that grows on purpose (see tak.toml): a pull request that | ||
| # regenerates it changes what `parse-usage` is, so its size is reported but not gated. | ||
| fixture_changed=false | ||
| if ! git diff --quiet "$base" -- benches/mise.usage.kdl benches/shadows/mise; then | ||
| fixture_changed=true | ||
| fi | ||
|
|
||
| failed=false | ||
| row() { | ||
| local label=$1 base_bytes=$2 head_bytes=$3 gated=$4 | ||
| local delta=$((head_bytes - base_bytes)) pct verdict | ||
| pct=$(awk -v b="$base_bytes" -v d="$delta" 'BEGIN { printf "%+.2f", b ? 100 * d / b : 0 }') | ||
| verdict=reported | ||
| if [ "$gated" = true ]; then | ||
| verdict=ok | ||
| # Against the exact ratio, not the rounded `pct`: +1.004% prints as +1.00% and still fails. | ||
| if awk -v b="$base_bytes" -v d="$delta" -v g="$GATE_PCT" 'BEGIN { exit !(b > 0 && 100 * d > g * b) }'; then | ||
| verdict="**over ${GATE_PCT}%**" | ||
| failed=true | ||
| fi | ||
| fi | ||
| printf '| %s | %s | %s | %+'"'"'d | %s%% | %s |\n' \ | ||
| "$label" "$(printf "%'d" "$base_bytes")" "$(printf "%'d" "$head_bytes")" "$delta" "$pct" "$verdict" | ||
| } | ||
|
|
||
| size_of() { bytes "$tmp/$1/$2"; } | ||
| share() { echo $(($(size_of "$1" parse-usage) - $(size_of "$1" parse-none))); } | ||
|
|
||
| shadow_gated=true | ||
| [ "$fixture_changed" = true ] && shadow_gated=false | ||
|
|
||
| report=$tmp/report.md | ||
| # The backticks below are markdown, not command substitution. | ||
| # shellcheck disable=SC2016 | ||
| { | ||
| printf '### Binary size\n\n' | ||
| printf 'Stripped release builds, `%s` against the merge base `%s`. ' "$(git rev-parse --short HEAD)" "$(git rev-parse --short "$base")" | ||
| printf 'Gated rows fail above +%s%%.\n\n' "$GATE_PCT" | ||
| printf '| binary | base | head | change | %% | gate |\n' | ||
| printf '|---|---:|---:|---:|---:|---|\n' | ||
| row '`usage` CLI' "$(size_of base usage)" "$(size_of head usage)" true | ||
| row 'derived mise CLI, usage'"'"'s share' "$(share base)" "$(share head)" "$shadow_gated" | ||
| row 'derived mise CLI, whole binary' "$(size_of base parse-usage)" "$(size_of head parse-usage)" false | ||
| row 'clap mise CLI, whole binary' "$(size_of base parse-clap)" "$(size_of head parse-clap)" false | ||
| printf '\n' | ||
| printf '"usage'"'"'s share" is `parse-usage` minus `parse-none`: the same program with and without the parse.\n' | ||
| if [ "$fixture_changed" = true ]; then | ||
| printf '\nThis change regenerates the mise shadow, so the derived CLI is reported rather than gated: its size moved with the fixture.\n' | ||
| fi | ||
| } >"$report" | ||
|
|
||
| cat "$report" >"$out" | ||
| [ "$failed" = false ] | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.