Skip to content

[ci-coach] Optimize integration test matrix by isolating slow Docker build test #11509

Closed

Description

@github-actions

CI Optimization: Reduce Integration Test Critical Path by 41%

Summary

This PR optimizes the integration test matrix by isolating the slow TestDockerBuild tests into a dedicated matrix group, reducing the critical path from 67.7s to ~40s (41% improvement).

Problem Analysis

The "CLI Completion & Other" catch-all group was the slowest integration test group at 67.7s, with a single test dominating execution time:

  • TestDockerBuild_WithMake: 38.43s (57% of group time)
  • Other Docker tests: ~0.4s combined
  • Total Docker tests: 3 tests, 38.4s

This bottleneck blocked the critical path, preventing faster test groups from completing earlier.

Solution

Created a new dedicated matrix group for Docker build tests:

  1. New group: "CLI Docker Build"

    • Pattern: TestDockerBuild
    • Expected duration: ~38-40s
    • Runs in parallel with other groups
  2. Updated "CLI Completion & Other" skip pattern

    • Added TestDockerBuild to exclusions
    • Reduced from 67.7s to ~29s (estimated)

Changes

File: .github/workflows/ci.yml

Added new matrix entry (after line 107):

- name: "CLI Docker Build"  # Isolate slow Docker build test (~38s)
  packages: "./pkg/cli"
  pattern: "TestDockerBuild"

Updated "CLI Completion & Other" skip pattern (line 111):

skip_pattern: "...|^TestAudit|^TestInspect|TestDockerBuild"

Expected Impact

Before:

  • Total matrix groups: 31
  • Critical path: 67.7s (CLI Completion & Other)
  • Docker tests block other test completion

After:

  • Total matrix groups: 32 (+1)
  • Critical path: ~40s (CLI Docker Build, runs in parallel)
  • Improvement: 41% faster critical path
  • Time saved: ~28 seconds per CI run

Cost-Benefit Analysis:

  • Impact: High (41% speedup on integration tests)
  • Effort: Low (4 line change, no logic changes)
  • Risk: Low (existing tests, just rebalancing execution)
  • Reversibility: High (easy to revert if issues arise)

Validation Results

✅ YAML syntax valid: Skip pattern correctly includes TestDockerBuild
✅ Matrix count: 32 groups (31 → 32, expected)
✅ CLI groups: 14 CLI test groups (13 → 14, expected)
✅ Pattern isolation: Docker tests will run in dedicated group

Note: Full validation with make lint && make build && make test-unit && make recompile cannot be performed in this workflow environment (no Go/Make available), but YAML structure has been verified.

Testing Plan

  • Verify workflow syntax passes GitHub Actions validation
  • Monitor first CI run after merge:
    • CLI Docker Build group should take ~38-40s
    • CLI Completion & Other should drop to ~25-30s
    • Critical path should be ~40s (down from 67.7s)
  • Compare with historical baselines from previous runs
  • Easy rollback: Revert this single commit if issues arise

Metrics Baseline (for comparison)

Current state (from CI run analysis):

  • Success rate: 71% (71 success, 26 cancelled, 3 action_required out of 100 runs)
  • Matrix groups: 31
  • Longest integration group: 67.7s (CLI Completion & Other)

Expected post-merge:

  • Matrix groups: 32
  • Longest integration group: ~40s (CLI Docker Build)
  • CLI Completion & Other: ~29s (reduced from 67.7s)
  • Time saved per run: ~28s

Analysis Methodology

This optimization was identified through systematic analysis of CI run data:

  1. Analyzed: 100 recent workflow runs
  2. Examined: Integration test artifacts with per-test timing from successful runs
  3. Identified: TestDockerBuild_WithMake taking 38.43s (57% of group time)
  4. Root cause: Single slow test in large catch-all group creating bottleneck
  5. Solution: Isolate slow tests into dedicated parallel group
View Detailed Test Timing Analysis

CLI Completion & Other Group (67.7s total)

Top 10 slowest tests:

  1. TestDockerBuild_WithMake: 38.43s ⚠️ (isolated by this PR)
  2. TestCollectWorkflowFiles_NoLockFile: 3.14s
  3. TestCompileWorkflowsWithNoEmit: 3.13s
  4. TestCompileWorkflows_TrialMode: 3.03s
  5. TestInteractiveWorkflowBuilder_compileWorkflow_SpinnerIntegration: 1.85s
  6. TestCompileWorkflows/nonexistent_specific_file: 1.62s
  7. TestCompileWorkflows: 1.62s
  8. TestDownloadRunArtifactsConcurrent_ConcurrencyLimit: 0.70s
  9. TestCompileWithFuzzyDailyScheduleDeterministic: 0.45s
  10. TestInstallPackage: 0.41s

Test grouping:

  • Docker: 3 tests, 38.4s (isolated by this PR)
  • Compile: 54 tests, 12.9s
  • Collect: 17 tests, 3.2s
  • Completion: 45 tests, 2.1s
  • Interactive: 17 tests, 1.9s
  • Other: ~2,300 tests, ~11s

After this PR:

  • CLI Docker Build: 3 tests, ~38-40s (new dedicated group)
  • CLI Completion & Other: 2,440 tests, ~29s (reduced from 67.7s)

References:

This optimization maintains 100% test coverage while improving CI efficiency through better workload distribution.

AI generated by CI Optimization Coach


Note

This was originally intended as a pull request, but the git push operation failed.

Workflow Run: View run details and download patch artifact

The patch file is available in the agent-artifacts artifact in the workflow run linked above.

To apply the patch locally:

# Download the artifact from the workflow run https://github.com/githubnext/gh-aw/actions/runs/21287478106
# (Use GitHub MCP tools if gh CLI is not available)
gh run download 21287478106 -n agent-artifacts

# The patch file will be at agent-artifacts/tmp/gh-aw/aw.patch after download
# Apply the patch
git am agent-artifacts/tmp/gh-aw/aw.patch
Show patch preview (52 of 52 lines)
From 0e700b6b2a7150a118c5d4558ac18a1c3dd0e397 Mon Sep 17 00:00:00 2001
From: CI Coach <ci-coach@github.com>
Date: Fri, 23 Jan 2026 13:29:17 +0000
Subject: [PATCH] ci: isolate slow Docker build test into dedicated matrix
 group
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Reduces integration test critical path from 67.7s to ~40s (41% improvement)

The 'CLI Completion & Other' catch-all group was the slowest at 67.7s,
with TestDockerBuild_WithMake taking 38.43s alone. This change:

- Creates dedicated 'CLI Docker Build' matrix group for Docker tests
- Updates 'CLI Completion & Other' skip pattern to exclude Docker tests
- Allows Docker tests to run in parallel, reducing critical path

Expected impact:
- Critical path: 67.7s → ~40s (41% faster)
- Time saved per CI run: ~28 seconds
- Matrix groups: 31 → 32 (minimal overhead)

Analysis based on CI run data showing 2,443 tests in the catch-all group
with Docker build test dominating execution time.
---
 .github/workflows/ci.yml | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 5a58a05..28be06e 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -105,10 +105,13 @@ jobs:
           - name: "CLI Audit & Inspect"
             packages: "./pkg/cli"
             pattern: "^TestAudit|^TestInspect"
+          - name: "CLI Docker Build"  # Isolate slow Docker build test (~38s)
+            packages: "./pkg/cli"
+            pattern: "TestDockerBuild"
           - name: "CLI Completion & Other"  # Remaining catch-all (reduced from original)
             packages: "./pkg/cli"
             pattern: ""  # Catch-all for tests not matched by other CLI patterns
-            skip_pattern: "^TestCompile[^W]|TestPoutine|TestMCPInspectPlaywright|TestMCPGateway|TestMCPAdd|TestMCPInspectGitHub|TestMCPServer|TestMCPConfig|TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|TestProgressFlagSignature
... (truncated)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions