Repository navigation
Description
CI Optimization: Reduce Integration Test Critical Path by 41%
Summary
This PR optimizes the integration test matrix by isolating the slow TestDockerBuild tests into a dedicated matrix group, reducing the critical path from 67.7s to ~40s (41% improvement).
Problem Analysis
The "CLI Completion & Other" catch-all group was the slowest integration test group at 67.7s, with a single test dominating execution time:
- TestDockerBuild_WithMake: 38.43s (57% of group time)
- Other Docker tests: ~0.4s combined
- Total Docker tests: 3 tests, 38.4s
This bottleneck blocked the critical path, preventing faster test groups from completing earlier.
Solution
Created a new dedicated matrix group for Docker build tests:
-
New group: "CLI Docker Build"
- Pattern:
TestDockerBuild - Expected duration: ~38-40s
- Runs in parallel with other groups
- Pattern:
-
Updated "CLI Completion & Other" skip pattern
- Added
TestDockerBuildto exclusions - Reduced from 67.7s to ~29s (estimated)
- Added
Changes
File: .github/workflows/ci.yml
Added new matrix entry (after line 107):
- name: "CLI Docker Build" # Isolate slow Docker build test (~38s)
packages: "./pkg/cli"
pattern: "TestDockerBuild"
Updated "CLI Completion & Other" skip pattern (line 111):
skip_pattern: "...|^TestAudit|^TestInspect|TestDockerBuild"
Expected Impact
Before:
- Total matrix groups: 31
- Critical path: 67.7s (CLI Completion & Other)
- Docker tests block other test completion
After:
- Total matrix groups: 32 (+1)
- Critical path: ~40s (CLI Docker Build, runs in parallel)
- Improvement: 41% faster critical path
- Time saved: ~28 seconds per CI run
Cost-Benefit Analysis:
- Impact: High (41% speedup on integration tests)
- Effort: Low (4 line change, no logic changes)
- Risk: Low (existing tests, just rebalancing execution)
- Reversibility: High (easy to revert if issues arise)
Validation Results
✅ YAML syntax valid: Skip pattern correctly includes TestDockerBuild
✅ Matrix count: 32 groups (31 → 32, expected)
✅ CLI groups: 14 CLI test groups (13 → 14, expected)
✅ Pattern isolation: Docker tests will run in dedicated group
Note: Full validation with make lint && make build && make test-unit && make recompile cannot be performed in this workflow environment (no Go/Make available), but YAML structure has been verified.
Testing Plan
- Verify workflow syntax passes GitHub Actions validation
- Monitor first CI run after merge:
- CLI Docker Build group should take ~38-40s
- CLI Completion & Other should drop to ~25-30s
- Critical path should be ~40s (down from 67.7s)
- Compare with historical baselines from previous runs
- Easy rollback: Revert this single commit if issues arise
Metrics Baseline (for comparison)
Current state (from CI run analysis):
- Success rate: 71% (71 success, 26 cancelled, 3 action_required out of 100 runs)
- Matrix groups: 31
- Longest integration group: 67.7s (CLI Completion & Other)
Expected post-merge:
- Matrix groups: 32
- Longest integration group: ~40s (CLI Docker Build)
- CLI Completion & Other: ~29s (reduced from 67.7s)
- Time saved per run: ~28s
Analysis Methodology
This optimization was identified through systematic analysis of CI run data:
- Analyzed: 100 recent workflow runs
- Examined: Integration test artifacts with per-test timing from successful runs
- Identified:
TestDockerBuild_WithMaketaking 38.43s (57% of group time) - Root cause: Single slow test in large catch-all group creating bottleneck
- Solution: Isolate slow tests into dedicated parallel group
View Detailed Test Timing Analysis
CLI Completion & Other Group (67.7s total)
Top 10 slowest tests:
- TestDockerBuild_WithMake: 38.43s
⚠️ (isolated by this PR) - TestCollectWorkflowFiles_NoLockFile: 3.14s
- TestCompileWorkflowsWithNoEmit: 3.13s
- TestCompileWorkflows_TrialMode: 3.03s
- TestInteractiveWorkflowBuilder_compileWorkflow_SpinnerIntegration: 1.85s
- TestCompileWorkflows/nonexistent_specific_file: 1.62s
- TestCompileWorkflows: 1.62s
- TestDownloadRunArtifactsConcurrent_ConcurrencyLimit: 0.70s
- TestCompileWithFuzzyDailyScheduleDeterministic: 0.45s
- TestInstallPackage: 0.41s
Test grouping:
- Docker: 3 tests, 38.4s (isolated by this PR)
- Compile: 54 tests, 12.9s
- Collect: 17 tests, 3.2s
- Completion: 45 tests, 2.1s
- Interactive: 17 tests, 1.9s
- Other: ~2,300 tests, ~11s
After this PR:
- CLI Docker Build: 3 tests, ~38-40s (new dedicated group)
- CLI Completion & Other: 2,440 tests, ~29s (reduced from 67.7s)
References:
- Analysis workflow run: §21287478106
This optimization maintains 100% test coverage while improving CI efficiency through better workload distribution.
AI generated by CI Optimization Coach
Note
This was originally intended as a pull request, but the git push operation failed.
Workflow Run: View run details and download patch artifact
The patch file is available in the agent-artifacts artifact in the workflow run linked above.
To apply the patch locally:
# Download the artifact from the workflow run https://github.com/githubnext/gh-aw/actions/runs/21287478106
# (Use GitHub MCP tools if gh CLI is not available)
gh run download 21287478106 -n agent-artifacts
# The patch file will be at agent-artifacts/tmp/gh-aw/aw.patch after download
# Apply the patch
git am agent-artifacts/tmp/gh-aw/aw.patch
Show patch preview (52 of 52 lines)
From 0e700b6b2a7150a118c5d4558ac18a1c3dd0e397 Mon Sep 17 00:00:00 2001
From: CI Coach <ci-coach@github.com>
Date: Fri, 23 Jan 2026 13:29:17 +0000
Subject: [PATCH] ci: isolate slow Docker build test into dedicated matrix
group
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Reduces integration test critical path from 67.7s to ~40s (41% improvement)
The 'CLI Completion & Other' catch-all group was the slowest at 67.7s,
with TestDockerBuild_WithMake taking 38.43s alone. This change:
- Creates dedicated 'CLI Docker Build' matrix group for Docker tests
- Updates 'CLI Completion & Other' skip pattern to exclude Docker tests
- Allows Docker tests to run in parallel, reducing critical path
Expected impact:
- Critical path: 67.7s → ~40s (41% faster)
- Time saved per CI run: ~28 seconds
- Matrix groups: 31 → 32 (minimal overhead)
Analysis based on CI run data showing 2,443 tests in the catch-all group
with Docker build test dominating execution time.
---
.github/workflows/ci.yml | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5a58a05..28be06e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -105,10 +105,13 @@ jobs:
- name: "CLI Audit & Inspect"
packages: "./pkg/cli"
pattern: "^TestAudit|^TestInspect"
+ - name: "CLI Docker Build" # Isolate slow Docker build test (~38s)
+ packages: "./pkg/cli"
+ pattern: "TestDockerBuild"
- name: "CLI Completion & Other" # Remaining catch-all (reduced from original)
packages: "./pkg/cli"
pattern: "" # Catch-all for tests not matched by other CLI patterns
- skip_pattern: "^TestCompile[^W]|TestPoutine|TestMCPInspectPlaywright|TestMCPGateway|TestMCPAdd|TestMCPInspectGitHub|TestMCPServer|TestMCPConfig|TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|TestProgressFlagSignature
... (truncated)
Activity