Skip to content

Pin iOS simulator version to match selected Xcode in provisioning - #34978

Merged
jfversluis merged 7 commits into
net11.0from
fix/pin-simulator-version
Apr 16, 2026
Merged

jfversluis merged 7 commits into
net11.0from
fix/pin-simulator-version

Conversation

@PureWeen

Copy link
Copy Markdown
Member

Problem

All 17 macOS CI jobs on net11.0 are failing after the SDK bump in #34852. The provisioning script runs xcodebuild -downloadPlatform iOS without pinning a version, which downloads the latest available simulator runtime (iOS 26.3.1) instead of the one matching the selected Xcode 26.2.

The macios SDK 26.2.11588-net11-p3 expects iphonesimulator SDK 23C53 (iOS 26.2), but the installed simulator is iOS 26.3.1 (23D8133). This causes:

actool: No simulator runtime version from ["22F77"] available to use with iphonesimulator SDK version 23C53
ibtool: iOS 26.2 Platform Not Installed

Evidence

From build 1379721 (Install Simulator Runtimes log):

  1. First attempt: Unable to connect to simulator (exit code 70)
  2. Script deletes all runtimes, retries
  3. Retry installs iOS 26.3.1 (not iOS 26.2)
  4. Every macOS build/test job fails downstream

The Xcode 26.0 case already had this fix — it explicitly pins -buildVersion 26.0. The bug was only in the else branch for Xcode 26.1+.

Fix

Pin -buildVersion ${MAJOR}.${MINOR} for all Xcode 26.x versions (where iOS version matches Xcode version). Include -architectureVariant universal for x64 simulator support on Apple Silicon agents. Keep unpinned download for pre-26 Xcode where the version mapping differs (Xcode 16.x → iOS 18.x).

Reviewed by

Approach validated by 3 independent AI models (Sonnet 4.6, Opus 4.6, Codex 5.3) — all agreed on the fix and flagged the two concerns addressed in the final version:

  1. ✅ Include -architectureVariant universal for all 26.x (not just 26.0)
  2. ✅ Guard with MAJOR == 26 to avoid breaking pre-26 Xcode branches

Copilot AI review requested due to automatic review settings April 15, 2026 13:17
@github-actions

github-actions Bot commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34978

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34978"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins the iOS simulator runtime download to the Xcode 26.x major/minor version during macOS provisioning to avoid installing a newer, incompatible simulator runtime and breaking downstream actool/ibtool steps in CI.

Changes:

  • Add a Xcode-26-wide branch that uses xcodebuild -downloadPlatform iOS -buildVersion ${MAJOR}.${MINOR}.
  • Ensure -architectureVariant universal is used for all Xcode 26.x simulator downloads.
  • Keep unpinned simulator downloads for Xcode < 26 where iOS/Xcode version mapping differs.

@PureWeen
PureWeen force-pushed the fix/pin-simulator-version branch from bbf7429 to f8da743 Compare April 15, 2026 16:11
@PureWeen

PureWeen commented Apr 15, 2026 •

Copy link
Copy Markdown
Member Author

Multi-Model Code Review — PR #34978

Pin iOS simulator version to match selected Xcode in provisioning

Reviewed by 3 independent reviewers with adversarial consensus across 5 review rounds.


Final Re-Review (after commit ce45cdb)

All Original Findings — Resolved ✅

# Severity Finding Status
1 🟡 MODERATE Xcode 27+ falls through to unpinned download ✅ FIXED — -ge "26"
2 🟡 MODERATE Merged step inherits continueOnError: true ✅ FIXED — removed from YAML, handled in-script
3 🟡 MODERATE Pipeline variables not persisted downstream ✅ FIXED — documented as intentional
4 🔴 CRITICAL echo ##vso[...] broken on bash ✅ FIXED — bash: echo "##vso[...]"
5 🟢 MINOR -architectureVariant universal without justification ✅ FIXED — comment added
6 🟡 MODERATE useExitCodeForErrors overrides continueOnError ✅ FIXED — nested under guard

New Finding (3/3 reviewers agree after adversarial consensus)

🔴 CRITICAL — useExitCodeForErrors is now dead code; test failures silently swallowed

File: eng/pipelines/common/run-dotnet-preview.yml

The error handling is gated on continueOnError -eq $false:

if ([System.Convert]::ToBoolean("${{ parameters.continueOnError }}") -eq $false -and $commandExitCode -ne 0) {
    # ... error logging and exit ...
}

But every caller that uses useExitCodeForErrors: true does NOT pass continueOnError — so it defaults to true:

Caller useExitCodeForErrors continueOnError Effective
stage-unit-tests.yml true not set true (default)
stage-integration-tests.yml true not set true (default)

Trace when tests fail (commandExitCode ≠ 0):

  1. Outer if: ToBoolean("true") -eq $false → $false → entire block skipped
  2. No error logged, no task.complete result=Failed, no exit
  3. Script exits 0 → step shows as Succeeded (full green)
  4. fail-on-issue.yml sees Succeeded (not SucceededWithIssues) → does nothing

Old behavior: Script called exit $LASTEXITCODE + emitted ##vso[task.complete result=Failed;]. YAML-level continueOnError: true softened this to SucceededWithIssues (yellow). fail-on-issue.yml then converted SucceededWithIssues → Failed. Test failures were visible.

New behavior: Test failures are completely invisible — no warning, no error, full green pipeline. This is a significant regression for CI observability.

Suggested fix — handle continueOnError=true + useExitCodeForErrors=true as a soft-fail (log error, mark task, but don't exit):

$commandExitCode = $LASTEXITCODE
if ($commandExitCode -ne 0) {
    if ([System.Convert]::ToBoolean("${{ parameters.continueOnError }}") -eq $false) {
        # Hard fail — exit immediately
        if ([System.Convert]::ToBoolean("${{ parameters.useExitCodeForErrors }}")) {
            Write-Host "##vso[task.logissue type=error]Test suite had $commandExitCode failure(s)."
        } else {
            Write-Host "##vso[task.logissue type=error]Command failed with exit code $commandExitCode."
        }
        Write-Host "##vso[task.complete result=Failed;]"
        exit $commandExitCode
    } elseif ([System.Convert]::ToBoolean("${{ parameters.useExitCodeForErrors }}")) {
        # Soft fail — log error + mark failed, but DON'T exit
        # (so downstream PublishTestResults steps still run)
        Write-Host "##vso[task.logissue type=error]Test suite had $commandExitCode failure(s)."
        Write-Host "##vso[task.complete result=Failed;]"
    }
}

This gives the correct behavior for all four parameter combinations:

continueOnError useExitCodeForErrors Command fails Result
false false yes Hard fail, generic message, exit ✅
false true yes Hard fail, test message, exit ✅
true false yes Silent continue (exits 0) ✅
true true yes Log error + mark SucceededWithIssues, no exit → PublishTestResults runs, fail-on-issue.yml catches it ✅

CI Status

Status Details
⏳ Pending maui-pr build not yet completed for latest commit
⚠️ Pre-existing AOT integration test failures — same on base branch

Recommendation: ⚠️ Request Changes

All 6 previous findings resolved ✅. One new critical issue: the in-script error handling needs an elseif branch for the continueOnError=true + useExitCodeForErrors=true case to preserve the old SucceededWithIssues behavior. Without it, test failures are invisible in CI.

@PureWeen

PureWeen commented Apr 15, 2026 •

Copy link
Copy Markdown
Member Author

/azp run maui-pr-uitests, maui-pr-devicetests

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

1 similar comment
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@jfversluis

Copy link
Copy Markdown
Member

/rebase

github-actions Bot and others added 6 commits April 16, 2026 08:16
Without -buildVersion, xcodebuild -downloadPlatform iOS downloads the
latest available simulator runtime, which may not match the selected
Xcode's SDK. On agents with Xcode 26.2, this installs iOS 26.3.1
instead of iOS 26.2, causing actool/ibtool failures:

  actool: No simulator runtime version from ["22F77"] available
  to use with iphonesimulator SDK version 23C53

  ibtool: iOS 26.2 Platform Not Installed

This broke all 17 macOS CI jobs on net11.0 after the SDK bump in
PR #34852.

Fix: Pin -buildVersion to ${MAJOR}.${MINOR} for Xcode 26.x (where
iOS version matches Xcode version). Include -architectureVariant
universal for x64 simulator support on Apple Silicon agents. Keep
unpinned download for pre-26 Xcode where version mapping differs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…variables

Merge the two PowerShell blocks in run-dotnet-preview.yml into one
(matching the pattern used by dotnet/android) so that DOTNET_ROOT,
DOTNET_MULTILEVEL_LOOKUP, and MSBuildEnableWorkloadResolver are
process-scoped only. Only PATH is persisted for subsequent steps.

Previously, persisting DOTNET_MULTILEVEL_LOOKUP=0 as a pipeline
variable poisoned subsequent pwsh: steps. System PowerShell 7.6
(Homebrew, framework-dependent) needs Microsoft.NETCore.App 10.0.0,
but with DOTNET_ROOT pointing to .dotnet/ (only .NET 11 preview)
and DOTNET_MULTILEVEL_LOOKUP=0 blocking fallback, pwsh crashed
with exit code 150 on all macOS integration test jobs.

Also convert a trivial pwsh: step to script: in setup-test-env.yml
as a defensive measure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fix echo quoting

- Change MAJOR == 26 to MAJOR -ge 26 so Xcode 27+ gets pinned
  simulator versions instead of silently falling through to unpinned
- Add 'Show simulator info' step (xcrun simctl runtime list/match)
  after simulator install for diagnostics (from rolfbjarne's PR #34979)
- Document -architectureVariant universal carry-forward from 26.0 block
- Fix echo quoting in setup-test-env.yml for Windows cmd.exe
  compatibility (cmd.exe prints surrounding quotes, breaking ##vso
  logging command parsing)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
script: runs bash where unquoted ## starts a comment, silently
dropping the ##vso logging command. Use bash: with quotes to ensure
the ##vso[task.setvariable] is printed correctly on all platforms.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the merged step inherited continueOnError: true from the
parameter default, which silently swallowed env setup failures (e.g.,
missing .dotnet/dotnet). Now continueOnError is handled inside the
script: env setup failures (PowerShell exceptions, missing binaries)
hard-fail the step, while the dotnet command respects the
continueOnError parameter for its exit code handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The useExitCodeForErrors block must be guarded by continueOnError=false
to preserve the original soft-fail semantics. Without the guard, test
failures hard-fail the step even when continueOnError=true (the
default), causing downstream PublishTestResults steps to be skipped
and test results never published.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
github-actions Bot force-pushed the fix/pin-simulator-version branch from ce45cdb to 37341dd Compare April 16, 2026 08:16
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

When continueOnError defaults to true (as in unit/integration test callers),
the error handling block was completely skipped, making test failures invisible.

Restructure to check commandExitCode first, then branch:
- continueOnError=false: hard fail with exit (unchanged)
- continueOnError=true + useExitCodeForErrors=true: soft fail — log error
  and mark task Failed but don't exit, so PublishTestResults still runs
  and fail-on-issue.yml catches the SucceededWithIssues status
- continueOnError=true + useExitCodeForErrors=false: silent continue (unchanged)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kubaflo

kubaflo commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Applied the suggested fix for the useExitCodeForErrors dead-code issue in run-dotnet-preview.yml (commit ac18dd1).

The error handling now checks $commandExitCode -ne 0 first, then branches on continueOnError:

continueOnError useExitCodeForErrors Failure behavior
false false Hard fail, generic message, exit
false true Hard fail, test count message, exit
true false Silent continue (exit 0)
true true Soft fail — logs error + marks task Failed, but does NOT exit so PublishTestResults runs and fail-on-issue.yml catches it

The last row was the broken case — previously the entire error block was skipped when continueOnError defaulted to true, silently swallowing test failures.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@jfversluis jfversluis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes what its supposed to

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants