Skip to content

ci(coverage): report test coverage from a workflow (#1063) #1064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JarryShaw merged 1 commit into from
Oct 6, 2026
Merged

ci(coverage): report test coverage from a workflow (#1063) #1064

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/coverage.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Coverage configuration for the one coverage-measured leg of unit-tests.yml
# (`test`, Python 3.14), passed with `--rcfile`. Coverage reads a single config
# file, so this repeats pyproject.toml's [tool.coverage.*] settings and adds the
# two the CI run needs on top. tests/project/test_coverage_rcfile.py asserts
# every pyproject setting is repeated here unchanged.
#
# parallel -- each process writes its own .coverage.* file, merged afterwards
# by `coverage combine`.
# patch -- "subprocess" (coverage 7.10+) starts measurement in every
# child Python process. pytest-xdist's workers are such children
# (execnet popen), so without it only the controller is measured
# and the -n auto run reports next to nothing.
# core -- "ctrace", not the sys.monitoring default on 3.12+. Thousands of
# tests purge and re-import pcapkit in setUp, and sysmon keeps
# state for every generation's code objects: measured on
# test_http_unit.py alone, peak RSS 0.40 GiB without coverage,
# 4.17 GiB under sysmon and 0.41 GiB under ctrace. Under
# -n auto that growth killed the runner (#1064).

[tool.coverage.run]
source = [
"pcapkit",
]
branch = true
parallel = true
patch = [
"subprocess",
]
core = "ctrace"

[tool.coverage.report]
show_missing = true
111 changes: 111 additions & 0 deletions .github/workflows/coverage-comment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: Coverage Comment

# #1063: posts the coverage the `test` job's Python 3.14 leg measured in
# unit-tests.yml as a pull-request comment. It runs no tests. The comment cannot
# be posted from unit-tests.yml itself: that file is also a reusable workflow,
# and a called workflow can only narrow the token its caller passes, never widen
# it. All four callers grant `contents: read`, so asking for
# `pull-requests: write` there would break every one of them, the release gate
# included.
#
# `workflow_run` runs this file as it is on the default branch, with a token of
# its own, so fork pull requests get a comment too. The price is that the
# triggering run's artifact is untrusted input. So this workflow:
#
# * never checks out or executes anything from the pull request;
# * takes the PR number from the API (the open PR whose head is this run's
# head sha), not from the artifact, and accepts it only if it is all digits;
# * renders its own Markdown from the artifact's numbers, accepting only the
# shapes the `test` job writes, rather than posting text from the run.
on:
workflow_run:
workflows: ["Unit Tests"]
types: [completed]

permissions:
actions: read
pull-requests: write

# One comment per PR, so serialise per head branch. A run that finishes after a
# newer push skips itself below, since its sha is no longer the PR's head.
concurrency:
group: coverage-comment-${{ github.event.workflow_run.head_repository.full_name }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: false

jobs:
comment:
name: Coverage comment
if: ${{ github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion != 'cancelled' }}
runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- name: Post or update the coverage comment
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
RUN_ID: ${{ github.event.workflow_run.id }}
RUN_URL: ${{ github.event.workflow_run.html_url }}
RUN_CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
HEAD_OWNER: ${{ github.event.workflow_run.head_repository.owner.login }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eu
case "$HEAD_SHA" in
*[!0-9a-f]*|'') echo "::error::unexpected head sha"; exit 1 ;;
esac

# `workflow_run.pull_requests` is empty for fork PRs, so ask the API.
pr=$(gh api -X GET "repos/$REPO/pulls" -f state=open -f head="$HEAD_OWNER:$HEAD_BRANCH" \
--jq ".[] | select(.head.sha == \"$HEAD_SHA\") | .number" | head -n 1)
case "$pr" in
*[!0-9]*|'') echo "::notice::No open PR has $HEAD_SHA as its head; nothing to comment on."; exit 0 ;;
esac

dir="$RUNNER_TEMP/coverage"
if ! gh run download "$RUN_ID" --repo "$REPO" --name coverage-summary --dir "$dir"; then
echo "::notice::Run $RUN_ID uploaded no coverage-summary artifact (the coverage leg did not get that far)."
exit 0
fi

python3 - "$dir/coverage-summary.json" > "$RUNNER_TEMP/body.md" <<'PY'
import json, os, re, sys
data = json.load(open(sys.argv[1]))
def number(value, ints=False):
ok = isinstance(value, int) if ints else isinstance(value, (int, float))
if isinstance(value, bool) or not ok or not 0 <= value < 10**7:
sys.exit(f'invalid number in artifact: {value!r}')
return value
tests = data['tests']
if tests not in ('success', 'failure', 'cancelled', 'skipped'):
sys.exit(f'invalid test outcome in artifact: {tests!r}')
total = number(data['total'])
status = '' if tests == 'success' else f' -- **tests {tests}**, so this is partial'
print(f"**Coverage: {total:.2f}%**{status} (unit tier, Python 3.14, "
f"`{os.environ['HEAD_SHA'][:9]}`, Unit Tests run {os.environ['RUN_CONCLUSION']})")
print()
print('| Package | Statements | Missed | Branches | Partial | Cover |')
print('|---|--:|--:|--:|--:|--:|')
for name, *counts, cover in data['packages']:
if not isinstance(name, str) or not re.fullmatch(r'pcapkit(/\w+| \(top level\))?', name):
sys.exit(f'invalid package name in artifact: {name!r}')
counts = [number(c, ints=True) for c in counts]
if len(counts) != 4:
sys.exit('invalid row in artifact')
print('| `{}` | {} | {} | {} | {} | {:.2f}% |'.format(name, *counts, number(cover)))
print()
print(f"Per-file detail: the `coverage-html` artifact of [this run]({os.environ['RUN_URL']}).")
print()
print('<!-- pcapkit-coverage-report -->')
PY

# Edit the one comment this workflow posted, found by its marker and
# author, rather than adding one per push.
id=$(gh api --paginate "repos/$REPO/issues/$pr/comments" --jq \
'.[] | select(.user.login == "github-actions[bot]" and (.body | contains("<!-- pcapkit-coverage-report -->"))) | .id' | head -n 1)
if [ -n "$id" ]; then
gh api -X PATCH "repos/$REPO/issues/comments/$id" -F body=@"$RUNNER_TEMP/body.md" > /dev/null
else
gh api -X POST "repos/$REPO/issues/$pr/comments" -F body=@"$RUNNER_TEMP/body.md" > /dev/null
fi
102 changes: 96 additions & 6 deletions .github/workflows/unit-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,15 @@ jobs:
- "3.12"
- "3.13"
- "3.14"
# #1063: exactly one leg runs under coverage, so measuring it adds no
# leg to the queue #1052 is cutting. 3.14 because it had the most
# headroom under the 45-minute cap on main (15.6, 15.8 and 16.8 minutes
# in runs 37393532176, 37392964804 and 37391296686, against 12.8-22.0
# for the other four), and because coverage's sys.monitoring core,
# which measures branches only from 3.14 on, is its cheapest tracer.
include:
- python-version: "3.14"
coverage: true

steps:
- uses: actions/checkout@v7
Expand Down Expand Up @@ -107,6 +116,12 @@ jobs:
python -m pip install -U pip setuptools wheel
python -m pip install -e '.[test,DPKT,crypto,NGAP]'

# 7.10 is the first release with `[run] patch = ["subprocess"]`, which
# .github/coverage.toml relies on to measure the xdist workers.
- name: Install coverage
if: ${{ matrix.coverage }}
run: python -m pip install 'coverage>=7.10'

# `-n auto` resolves to whatever this runner reports, and nothing else in
# the log says what that was -- print it so the number `-n auto` picked
# is a measured fact instead of an assumption about runner size.
Expand All @@ -116,15 +131,90 @@ jobs:
python -c "import os; print('cpu_count', os.cpu_count())"

- name: Run unit tests
id: tests
# `--dist load` is pinned explicitly rather than left to default: a
# future change to xdist's own default, or someone "optimising" to
# `loadfile`/`loadscope`, cannot silently change how this suite is
# distributed.
run: >-
python -m pytest -q -n auto --dist load
--ignore=tests/integration
--ignore-glob='*_runtime.py'
--ignore-glob='*_regression.py'
# distributed. The coverage leg runs the identical selection.
env:
COVERAGE: ${{ matrix.coverage }}
run: |
set -- -q -n auto --dist load \
--ignore=tests/integration \
--ignore-glob='*_runtime.py' \
--ignore-glob='*_regression.py'
if [ "$COVERAGE" = "true" ]; then
python -m coverage run --rcfile=.github/coverage.toml -m pytest "$@"
else
python -m pytest "$@"
fi

# The table goes to the job summary. The numbers also go to an artifact
# that coverage-comment.yml renders as a PR comment once this run
# completes; that comment cannot be posted from here, because this file is
# also a reusable workflow and its callers grant only `contents: read`.
# `!cancelled()` so a failing test still reports what it measured, and
# says the tests failed in the first line.
- name: Report coverage
if: ${{ !cancelled() && matrix.coverage }}
shell: bash
env:
SHA: ${{ github.sha }}
TESTS_OUTCOME: ${{ steps.tests.outcome }}
run: |
set -euo pipefail
python -m coverage combine --rcfile=.github/coverage.toml
python -m coverage report --rcfile=.github/coverage.toml
python -m coverage html --rcfile=.github/coverage.toml -d coverage-html
python -m coverage json --rcfile=.github/coverage.toml -o coverage.json
mkdir -p coverage-comment
# Writes the numbers as JSON for coverage-comment.yml, which renders
# its own Markdown from them rather than posting text from this run.
python - <<'PY' >> "$GITHUB_STEP_SUMMARY"
import collections, json, os
data = json.load(open('coverage.json'))
keys = ('num_statements', 'missing_lines', 'num_branches', 'num_partial_branches',
'covered_lines', 'covered_branches')
packages = collections.defaultdict(collections.Counter)
for path, info in data['files'].items():
parts = path.split('/')
name = '/'.join(parts[:2]) if len(parts) > 2 else 'pcapkit (top level)'
packages[name].update({k: info['summary'][k] for k in keys})
def pct(c):
total = c['num_statements'] + c['num_branches']
return 100.0 * (c['covered_lines'] + c['covered_branches']) / total if total else 100.0
rows = [[name, c['num_statements'], c['missing_lines'], c['num_branches'],
c['num_partial_branches'], round(pct(c), 2)] for name, c in sorted(packages.items())]
total = round(data['totals']['percent_covered'], 2)
tests = os.environ['TESTS_OUTCOME']
json.dump({'total': total, 'tests': tests, 'packages': rows},
open('coverage-comment/coverage-summary.json', 'w'))
status = '' if tests == 'success' else f' -- **tests {tests}**, so this is partial'
print(f"**Coverage: {total:.2f}%**{status} (unit tier, Python 3.14, `{os.environ['SHA'][:9]}`)")
print()
print('| Package | Statements | Missed | Branches | Partial | Cover |')
print('|---|--:|--:|--:|--:|--:|')
for row in rows:
print('| `{}` | {} | {} | {} | {} | {:.2f}% |'.format(*row))
print()
print('Per-file detail: the `coverage-html` artifact of this run.')
PY

- name: Upload coverage HTML report
if: ${{ !cancelled() && matrix.coverage }}
uses: actions/upload-artifact@v7
with:
name: coverage-html
path: coverage-html/
retention-days: 30

- name: Upload coverage summary
if: ${{ !cancelled() && matrix.coverage }}
uses: actions/upload-artifact@v7
with:
name: coverage-summary
path: coverage-comment/coverage-summary.json
retention-days: 7

integration:
name: Integration Python ${{ matrix.python-version }}
Expand Down
4 changes: 2 additions & 2 deletions docs/source/contributing/releasing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Editing ``__version__`` by hand without also moving :file:`CITATION.cff`'s
(``:494-514``) asserts the two agree, and ``create-release.yml``'s
``unit-tests`` job (``:362-376``) calls ``unit-tests.yml`` with
``gate-only: true``, which runs the **full** suite rather than the tiered
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:989-1058``).
subset an ordinary push runs (its ``gate`` job, ``unit-tests.yml:1079-1148``).
That gate runs whenever a release will -- a moved ``__version__`` is exactly
what makes ``version_check``'s evidence read "not yet published" -- and every
publishing job requires it to have succeeded (``:392,493,573,674``), so a
Expand All @@ -56,7 +56,7 @@ Either run the script, or move both fields by hand in the same commit.
This is the **only** edit a person makes to get a release started -- the tag,
the Release, and every upload are the workflow's job from here. Two other
checks run unconditionally on the same path, though neither is a step in *this*
process: the ``changelog`` job (``unit-tests.yml:962-983``) fails outright if
process: the ``changelog`` job (``unit-tests.yml:1052-1073``) fails outright if
``CHANGELOG.md`` has drifted from its source entry under
:file:`docs/source/changelog/`, and the release-body step warns, without
failing, if that entry's heading still reads "unreleased"
Expand Down
Loading
Loading