Repository navigation
test: run the bats suites on macOS with scripts/test-local.sh #2535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
0c957ea
4291bce
e9f3cdb
9ca318f
899f14a
0a33bff
c922fe5
a6299f4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # Brewfile — the GNU toolchain the scripts and bats suites assume, for macOS. | ||
| # | ||
| # CI runs on ubuntu-latest. macOS ships bash 3.2 and BSD sed, grep, date, find | ||
| # and awk, under which about a third of the bats tests fail for reasons that are | ||
| # not bugs. Install these once, then run the suites through scripts/test-local.sh, | ||
| # which puts them first on PATH for that run only: | ||
| # | ||
| # brew bundle --file=Brewfile | ||
| # scripts/test-local.sh | ||
|
|
||
| brew "bash" # bash 5: declare -g/-A, mapfile, ${var^^} | ||
| brew "bats-core" | ||
| brew "coreutils" # GNU date -d, base64 -w, stat -c, mktemp, timeout | ||
| brew "gnu-sed" # sed -i without a suffix, sed -z | ||
| brew "grep" # grep -P | ||
| brew "findutils" # GNU find and xargs | ||
| brew "gawk" # multi-line -v values; BSD awk fails with "newline in string" | ||
| brew "flock" # scripts/lib/miss-harvest.sh, scripts/lib/token-metrics.sh | ||
| brew "parallel" # only for scripts/test-local.sh --jobs | ||
| brew "jq" | ||
| brew "yq" # mikefarah yq v4, the one ubuntu-latest ships | ||
| brew "python@3.12" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| # test-local.sh — run the bats suites on a developer machine with the toolchain | ||
| # CI has. | ||
| # | ||
| # CI runs on ubuntu-latest: GNU bash 5, coreutils, sed, grep, findutils, gawk, | ||
| # flock, and a python3 with jsonschema and PyYAML. macOS has bash 3.2 and the BSD | ||
| # tools instead, and a plain `bats` run there fails about 2,000 tests that pass | ||
| # in CI (measured on main, 2026-11-13). This script, on macOS: | ||
| # | ||
| # 1. puts the Homebrew GNU tools and bash 5 first on PATH, for this run only, | ||
| # and fails naming the missing formula when one is not installed | ||
| # (brew bundle --file=Brewfile installs them all); | ||
| # 2. resolves TMPDIR to its real path (/var is a symlink to /private/var, and | ||
| # a test comparing $PWD with a path built from TMPDIR fails otherwise); | ||
| # 3. on any OS, builds a cached venv from the hash-pinned requirements/ci/ | ||
| # files when python3 is older than 3.10 or cannot import jsonschema and | ||
| # yaml (the validator suites need both), and puts it first on PATH. | ||
| # | ||
| # Then it runs bats. With no test paths it runs every suite: the top-level | ||
| # tests/*.bats, tests/dev-lead/unit and tests/dev-lead/integration, which is a | ||
| # superset of what lint.yml and test-dev-lead.yml run. | ||
| # | ||
| # Usage: scripts/test-local.sh [--jobs N] [bats options] [test paths...] | ||
| # --jobs N run N test files at a time (needs GNU parallel). Tests inside a | ||
| # file still run one at a time. Default: 1. | ||
| # Any other option is passed to bats, e.g. -f <regex>, --tap. | ||
| # | ||
| # This script itself has to start under macOS bash 3.2: no bash 4 syntax above | ||
| # the exec. | ||
|
|
||
| repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | ||
| cd "$repo_root" | ||
|
|
||
| jobs=1 | ||
| bats_args=() | ||
| paths=() | ||
| while [ "$#" -gt 0 ]; do | ||
|
Check failure on line 38 in scripts/test-local.sh
|
||
| case "$1" in | ||
| --jobs|-j) jobs="${2:?--jobs needs a number}"; shift 2 ;; | ||
| --jobs=*) jobs="${1#--jobs=}"; shift ;; | ||
| -h|--help) sed -n '3,30p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; | ||
| -f|--filter|--negative-filter|-F|--formatter|--report-formatter|--parallel-binary-name|--filter-status|--filter-tags|-o|--output|--gather-test-outputs-in|--line-reference-format|--code-quote-style) bats_args+=("$1" "${2:?$1 needs a value}"); shift 2 ;; | ||
| -*) bats_args+=("$1"); shift ;; | ||
| *) paths+=("$1"); shift ;; | ||
| esac | ||
| done | ||
| if [ "${#paths[@]}" -eq 0 ]; then | ||
|
Check failure on line 48 in scripts/test-local.sh
|
||
| paths=(tests tests/dev-lead/unit tests/dev-lead/integration) | ||
| fi | ||
|
|
||
| die() { echo "test-local: $*" >&2; exit 1; } | ||
|
|
||
| if [ "$(uname -s)" = Darwin ]; then | ||
|
Check failure on line 54 in scripts/test-local.sh
|
||
| if [ -n "${HOMEBREW_PREFIX:-}" ]; then | ||
|
Check failure on line 55 in scripts/test-local.sh
|
||
| brew_prefix="$HOMEBREW_PREFIX" | ||
| elif command -v brew >/dev/null 2>&1; then | ||
| brew_prefix="$(brew --prefix)" | ||
| else | ||
| die "Homebrew not found. Install it from https://brew.sh, then run: brew bundle --file=Brewfile" | ||
| fi | ||
|
|
||
| missing="" | ||
| gnu_path="" | ||
| for formula in coreutils gnu-sed grep findutils gawk; do | ||
|
codeant-ai[bot] marked this conversation as resolved.
Resolved
|
||
| dir="$brew_prefix/opt/$formula/libexec/gnubin" | ||
| if [ -d "$dir" ]; then | ||
|
Check failure on line 67 in scripts/test-local.sh
|
||
| gnu_path="${gnu_path}${dir}:" | ||
| else | ||
| missing="$missing $formula" | ||
| fi | ||
| done | ||
| bash5="$brew_prefix/bin/bash" | ||
| if [ ! -x "$bash5" ] || ! "$bash5" -c '[ "${BASH_VERSINFO[0]}" -ge 4 ]'; then | ||
|
Check failure on line 74 in scripts/test-local.sh
|
||
| missing="$missing bash" | ||
| fi | ||
| [ -x "$brew_prefix/bin/bats" ] || missing="$missing bats-core" | ||
|
Check failure on line 77 in scripts/test-local.sh
|
||
| [ -x "$brew_prefix/bin/flock" ] || missing="$missing flock" | ||
|
Check failure on line 78 in scripts/test-local.sh
|
||
|
don-petry marked this conversation as resolved.
Resolved
|
||
| # macOS may ship its own jq in /usr/bin; any jq and yq on PATH will do. | ||
| command -v jq >/dev/null 2>&1 || [ -x "$brew_prefix/bin/jq" ] || missing="$missing jq" | ||
|
Check failure on line 80 in scripts/test-local.sh
|
||
| command -v yq >/dev/null 2>&1 || [ -x "$brew_prefix/bin/yq" ] || missing="$missing yq" | ||
|
Check failure on line 81 in scripts/test-local.sh
|
||
| if [ "$jobs" != 1 ] && [ ! -x "$brew_prefix/bin/parallel" ]; then | ||
|
Check failure on line 82 in scripts/test-local.sh
|
||
| missing="$missing parallel" | ||
| fi | ||
| if [ -n "$missing" ]; then | ||
|
Check failure on line 85 in scripts/test-local.sh
|
||
| die "missing Homebrew formulae:${missing}. Run: brew bundle --file=Brewfile" | ||
| fi | ||
|
|
||
| PATH="${gnu_path}${brew_prefix}/bin:$PATH" | ||
| TMPDIR="$(cd "${TMPDIR:-/tmp}" 2>/dev/null && pwd -P)" || TMPDIR="$(cd /tmp && pwd -P)" | ||
| export PATH TMPDIR | ||
| fi | ||
|
|
||
| # Python deps for the validator suites. Use the python3 on PATH when it is new | ||
| # enough and already has them (CI installs them system-wide). macOS's own 3.9 | ||
| # can import both and still fail the validators, whose annotations need 3.10. | ||
| # Otherwise build a venv once per requirements change, outside the checkout so | ||
| # every worktree shares it. | ||
| if ! python3 -c 'import sys, jsonschema, yaml; sys.exit(sys.version_info < (3, 10))' >/dev/null 2>&1; then | ||
| py="" | ||
| for candidate in python3.12 python3.13 python3; do | ||
| if command -v "$candidate" >/dev/null 2>&1 \ | ||
| && "$candidate" -c 'import sys; sys.exit(sys.version_info < (3, 10))' 2>/dev/null; then | ||
| py="$candidate" | ||
| break | ||
| fi | ||
| done | ||
| [ -n "$py" ] || die "no python3 >= 3.10 found for the validator suites. Run: brew bundle --file=Brewfile" | ||
|
Check failure on line 108 in scripts/test-local.sh
|
||
|
|
||
| reqs=(requirements/ci/jsonschema.txt requirements/ci/pyyaml.txt requirements/ci/yamllint.txt) | ||
| venv_root="${COMBSMITH_TEST_VENV:-${XDG_CACHE_HOME:-$HOME/.cache}/combsmith-github-private/test-venv}" | ||
| if command -v sha256sum >/dev/null 2>&1; then sha=(sha256sum); else sha=(shasum -a 256); fi | ||
| stamp="$( { "$py" --version; cat "${reqs[@]}"; } | "${sha[@]}" | cut -d' ' -f1)" | ||
| # One venv per interpreter + requirements, so a worktree on other | ||
| # requirements never replaces a venv another run is using. Concurrent runs | ||
| # that need the same new venv build it once, under a lock; flock is required | ||
| # for that (checked on macOS above, util-linux on Linux). | ||
| venv="$venv_root/$stamp" | ||
| mkdir -p "$venv_root" | ||
| if [ ! -f "$venv/.complete" ]; then | ||
|
Check failure on line 120 in scripts/test-local.sh
|
||
| command -v flock >/dev/null 2>&1 \ | ||
| || die "flock is needed to build the python venv safely (macOS: brew bundle --file=Brewfile; Linux: util-linux)" | ||
| exec 9>"$venv.lock" | ||
| flock 9 | ||
| if [ ! -f "$venv/.complete" ]; then | ||
|
Check failure on line 125 in scripts/test-local.sh
|
||
| echo "test-local: building the python venv at $venv" >&2 | ||
| rm -rf "$venv" | ||
| "$py" -m venv "$venv" | ||
| "$venv/bin/python" -m pip install --quiet --disable-pip-version-check --require-hashes --only-binary ":all:" \ | ||
| -r requirements/ci/jsonschema.txt -r requirements/ci/pyyaml.txt -r requirements/ci/yamllint.txt \ | ||
| || die "pip install into $venv failed" | ||
| : > "$venv/.complete" | ||
| fi | ||
| exec 9>&- | ||
| fi | ||
| PATH="$venv/bin:$PATH" | ||
| export PATH | ||
| fi | ||
|
|
||
| if [ "$jobs" != 1 ]; then | ||
|
Check failure on line 140 in scripts/test-local.sh
|
||
| bats_args+=(--jobs "$jobs" --no-parallelize-within-files) | ||
| fi | ||
|
|
||
| # Close stdin: a test that reads it would otherwise wait on the terminal forever. | ||
| exec bats ${bats_args[@]+"${bats_args[@]}"} "${paths[@]}" </dev/null | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| #!/usr/bin/env bats | ||
| # Unit tests for toolchain_preflight.bash — the macOS toolchain check every | ||
| # suite-level setup_suite runs. | ||
|
don-petry marked this conversation as resolved.
Resolved
|
||
| # | ||
| # Each test puts stubs for uname and the checked tools first on PATH, so the | ||
| # result does not depend on the machine running it. | ||
|
|
||
| load toolchain_preflight | ||
|
|
||
| setup() { | ||
| STUB_BIN_DIR="$BATS_TEST_TMPDIR/bin" | ||
| mkdir -p "$STUB_BIN_DIR" | ||
| export PATH="$STUB_BIN_DIR:$PATH" | ||
| unset BATS_SKIP_TOOLCHAIN_PREFLIGHT | ||
| # A GNU toolchain on macOS. grep is the real one: GNU on CI and under | ||
| # scripts/test-local.sh, which are where this suite runs. | ||
| _stub uname 'echo Darwin' | ||
| _stub sed 'echo "sed (GNU sed) 4.10"' | ||
| _stub date 'exit 0' | ||
| _stub awk 'echo "GNU Awk 5.3.2"' | ||
| _stub flock 'exit 0' | ||
| _stub python3 'exit 0' | ||
| } | ||
|
|
||
| # _stub <name> <body> — a command on PATH that runs <body>. The shebang names | ||
| # this bash by path, so a stub still runs when PATH holds only the stub dir. | ||
| _stub() { | ||
| printf '#!%s\n%s\n' "$BASH" "$2" > "$STUB_BIN_DIR/$1" | ||
| chmod +x "$STUB_BIN_DIR/$1" | ||
| } | ||
|
|
||
| _preflight_with_only_stubs_on_path() { | ||
| local PATH="$STUB_BIN_DIR" | ||
| toolchain_preflight | ||
| } | ||
|
|
||
| @test "macOS with the GNU toolchain passes" { | ||
| run toolchain_preflight | ||
| [ "$status" -eq 0 ] | ||
| [ -z "$output" ] | ||
| } | ||
|
|
||
| @test "macOS with BSD sed fails, names sed and points to scripts/test-local.sh" { | ||
| _stub sed 'echo "usage: sed script [-EHalnru] [-i extension] [file ...]" >&2; exit 1' | ||
| run toolchain_preflight | ||
| [ "$status" -eq 1 ] | ||
| [[ "$output" == *"GNU sed"* ]] | ||
| [[ "$output" == *"scripts/test-local.sh"* ]] | ||
| [[ "$output" != *"flock"* ]] | ||
| } | ||
|
|
||
| @test "macOS without flock or the python deps names both" { | ||
| rm "$STUB_BIN_DIR/flock" | ||
| _stub python3 'exit 1' | ||
| # Only the stubs on PATH, so a real flock (/usr/bin/flock on Ubuntu) cannot | ||
| # satisfy the check; grep is passed through to the real one. | ||
| _stub grep "exec $(command -v grep) \"\$@\"" | ||
| run _preflight_with_only_stubs_on_path | ||
| [ "$status" -eq 1 ] | ||
| [[ "$output" == *"flock is not installed"* ]] | ||
| [[ "$output" == *"jsonschema"* ]] | ||
| } | ||
|
|
||
| @test "macOS with BATS_SKIP_TOOLCHAIN_PREFLIGHT=1 skips the check" { | ||
| _stub sed 'exit 1' | ||
| BATS_SKIP_TOOLCHAIN_PREFLIGHT=1 run toolchain_preflight | ||
| [ "$status" -eq 0 ] | ||
| } | ||
|
|
||
| @test "Linux is not checked, whatever the tools" { | ||
| _stub uname 'echo Linux' | ||
| _stub sed 'exit 1' | ||
| _stub date 'exit 1' | ||
| run toolchain_preflight | ||
| [ "$status" -eq 0 ] | ||
| } | ||
|
|
||
| @test "macOS: a TMPDIR that does not exist falls back to /tmp" { | ||
| TMPDIR="$BATS_TEST_TMPDIR/missing" | ||
| toolchain_preflight | ||
| [ "$TMPDIR" = "$(cd /tmp && pwd -P)" ] | ||
| } | ||
|
|
||
| @test "macOS: TMPDIR is resolved to its real path" { | ||
| mkdir -p "$BATS_TEST_TMPDIR/real" | ||
| ln -s "$BATS_TEST_TMPDIR/real" "$BATS_TEST_TMPDIR/link" | ||
| TMPDIR="$BATS_TEST_TMPDIR/link" | ||
| toolchain_preflight | ||
| [ "$TMPDIR" = "$(cd "$BATS_TEST_TMPDIR/real" && pwd -P)" ] | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| #!/usr/bin/env bash | ||
| # toolchain_preflight.bash — on macOS, stop a bats run that would fail for want | ||
| # of the GNU toolchain, and say what to run instead. | ||
| # | ||
| # The scripts under test assume what ubuntu-latest has: bash 5, GNU sed, grep, | ||
| # date, gawk, flock, and a python3 with jsonschema and PyYAML. Under stock macOS | ||
| # tools a run fails about 2,000 tests with messages like `declare: -g: invalid | ||
| # option` and `sed: illegal option -- z`, which read as real failures. This | ||
| # check turns that into one line naming the missing tool and the fix: | ||
| # scripts/test-local.sh, which sets the toolchain up for the run. | ||
| # | ||
| # Only macOS is checked; CI and Linux machines are untouched. It runs inside | ||
| # bats, so it may be under bash 3.2 itself: no bash 4 syntax here. | ||
| # BATS_SKIP_TOOLCHAIN_PREFLIGHT=1 skips it, for a deliberate run of a suite | ||
| # that does not need the toolchain. | ||
| # | ||
| # Also resolves TMPDIR to its real path (/var is a symlink to /private/var), so | ||
| # a path built from TMPDIR matches $PWD after a cd into it. | ||
|
|
||
| toolchain_preflight() { | ||
| [ "$(uname -s)" = Darwin ] || return 0 | ||
|
Check failure on line 21 in tests/dev-lead/unit/toolchain_preflight.bash
|
||
| [ "${BATS_SKIP_TOOLCHAIN_PREFLIGHT:-}" = 1 ] && return 0 | ||
|
Check failure on line 22 in tests/dev-lead/unit/toolchain_preflight.bash
|
||
|
|
||
| local problems="" | ||
| [ "${BASH_VERSINFO[0]}" -ge 4 ] || problems="${problems} | ||
|
Check failure on line 25 in tests/dev-lead/unit/toolchain_preflight.bash
|
||
|
don-petry marked this conversation as resolved.
Resolved
|
||
| - bash ${BASH_VERSION} runs bats; the scripts need bash 4 or later" | ||
| sed --version 2>/dev/null | grep -q GNU || problems="${problems} | ||
| - sed is BSD sed; the tests need GNU sed (sed -i, sed -z)" | ||
| # Not `grep --version`: BSD grep calls itself "GNU compatible" there. | ||
| echo x | grep -qP 'x' 2>/dev/null || problems="${problems} | ||
| - grep is BSD grep; the scripts need GNU grep (grep -P)" | ||
| date -d @0 >/dev/null 2>&1 || problems="${problems} | ||
| - date is BSD date; the scripts need GNU date (date -d)" | ||
| awk --version 2>/dev/null | grep -q GNU || problems="${problems} | ||
| - awk is BSD awk; the scripts need gawk" | ||
| command -v flock >/dev/null 2>&1 || problems="${problems} | ||
| - flock is not installed" | ||
| python3 -c 'import sys, jsonschema, yaml; sys.exit(sys.version_info < (3, 10))' >/dev/null 2>&1 || problems="${problems} | ||
| - python3 is older than 3.10 or cannot import jsonschema and yaml (the validator suites need both)" | ||
|
|
||
| if [ -n "$problems" ]; then | ||
|
Check failure on line 41 in tests/dev-lead/unit/toolchain_preflight.bash
|
||
| printf 'toolchain preflight: this run would fail for reasons that are not bugs:%s\n' "$problems" >&2 | ||
| printf 'Run the suites with scripts/test-local.sh instead (first time: brew bundle --file=Brewfile).\n' >&2 | ||
| printf 'Set BATS_SKIP_TOOLCHAIN_PREFLIGHT=1 to run anyway.\n' >&2 | ||
| return 1 | ||
| fi | ||
|
|
||
| TMPDIR="$(cd "${TMPDIR:-/tmp}" 2>/dev/null && pwd -P)" || TMPDIR="$(cd /tmp && pwd -P)" | ||
| export TMPDIR | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.