Skip to content

[perf-improver] Track performance regressions in CI: integrate MSTest.Performance.Runner timing output #9312

Closed

Description

Problem

MSTest.Performance.Runner is a capable local profiling harness (PerfView, dotnet-trace, timing via PlainProcess), but it has no CI integration. Performance regressions can be merged silently because:

  • No automated timing baseline is checked on every PR or nightly.
  • There is no trend tracking (no history of "how long does Scenario1 take to run?").

Proposal

Add a lightweight CI step — ideally on a Linux runner since it's the most reproducible — that:

  1. Runs ./build.sh -pack to produce the MSTest NuGet packages.
  2. Runs MSTest.Performance.Runner execute --pipelineNameFilter "*PlainProcess*" to collect wall-clock and CPU-time data for both MethodLevel and ClassLevel scenarios (now cross-platform after [perf-improver] Extend performance runner timing scenarios to Linux/macOS and add ClassLevel variant #9311).
  3. Uploads Result.json as a CI artifact for trend tracking.
  4. (Optional, phase 2) Compares the timing against a stored baseline and fails the check if regression exceeds a threshold (e.g. >10% slowdown).

Phase 1 (artifact-only) is low-risk: it does not gate PRs and only adds visibility. Phase 2 (regression gate) should be discussed with maintainers before implementation.

Why now

PR #9311 removes the Windows-only restriction from the timing pipelines, making a Linux CI runner a viable host for the first time. This issue tracks the follow-up work to wire it into the CI pipeline.

Effort estimate

  • Phase 1: small — a new GitHub Actions job, ~1 day of work.
  • Phase 2: medium — requires agreed baseline values and acceptable-regression thresholds.

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.3K AIC · ⌖ 25.8 AIC · ⊞ 57.6K · [◷]( · ◷)

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/performanceRuntime / build performance / efficiency.type/automationCreated or maintained by an agentic workflow.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions