Repository navigation
Pin iOS simulator version to match selected Xcode in provisioning - #34978
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34978Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34978" |
There was a problem hiding this comment.
Pull request overview
Pins the iOS simulator runtime download to the Xcode 26.x major/minor version during macOS provisioning to avoid installing a newer, incompatible simulator runtime and breaking downstream actool/ibtool steps in CI.
Changes:
- Add a Xcode-26-wide branch that uses
xcodebuild -downloadPlatform iOS -buildVersion ${MAJOR}.${MINOR}. - Ensure
-architectureVariant universalis used for all Xcode 26.x simulator downloads. - Keep unpinned simulator downloads for Xcode < 26 where iOS/Xcode version mapping differs.
bbf7429 to
f8da743
Compare
Multi-Model Code Review — PR #34978Pin iOS simulator version to match selected Xcode in provisioning Reviewed by 3 independent reviewers with adversarial consensus across 5 review rounds. Final Re-Review (after commit ce45cdb)All Original Findings — Resolved ✅
New Finding (3/3 reviewers agree after adversarial consensus)🔴 CRITICAL —
|
| Caller | useExitCodeForErrors |
continueOnError |
Effective |
|---|---|---|---|
stage-unit-tests.yml |
true |
not set | true (default) |
stage-integration-tests.yml |
true |
not set | true (default) |
Trace when tests fail (commandExitCode ≠ 0):
- Outer
if:ToBoolean("true") -eq $false→$false→ entire block skipped - No error logged, no
task.complete result=Failed, noexit - Script exits 0 → step shows as Succeeded (full green)
fail-on-issue.ymlseesSucceeded(notSucceededWithIssues) → does nothing
Old behavior: Script called exit $LASTEXITCODE + emitted ##vso[task.complete result=Failed;]. YAML-level continueOnError: true softened this to SucceededWithIssues (yellow). fail-on-issue.yml then converted SucceededWithIssues → Failed. Test failures were visible.
New behavior: Test failures are completely invisible — no warning, no error, full green pipeline. This is a significant regression for CI observability.
Suggested fix — handle continueOnError=true + useExitCodeForErrors=true as a soft-fail (log error, mark task, but don't exit):
$commandExitCode = $LASTEXITCODE
if ($commandExitCode -ne 0) {
if ([System.Convert]::ToBoolean("${{ parameters.continueOnError }}") -eq $false) {
# Hard fail — exit immediately
if ([System.Convert]::ToBoolean("${{ parameters.useExitCodeForErrors }}")) {
Write-Host "##vso[task.logissue type=error]Test suite had $commandExitCode failure(s)."
} else {
Write-Host "##vso[task.logissue type=error]Command failed with exit code $commandExitCode."
}
Write-Host "##vso[task.complete result=Failed;]"
exit $commandExitCode
} elseif ([System.Convert]::ToBoolean("${{ parameters.useExitCodeForErrors }}")) {
# Soft fail — log error + mark failed, but DON'T exit
# (so downstream PublishTestResults steps still run)
Write-Host "##vso[task.logissue type=error]Test suite had $commandExitCode failure(s)."
Write-Host "##vso[task.complete result=Failed;]"
}
}This gives the correct behavior for all four parameter combinations:
continueOnError |
useExitCodeForErrors |
Command fails | Result |
|---|---|---|---|
false |
false |
yes | Hard fail, generic message, exit ✅ |
false |
true |
yes | Hard fail, test message, exit ✅ |
true |
false |
yes | Silent continue (exits 0) ✅ |
true |
true |
yes | Log error + mark SucceededWithIssues, no exit → PublishTestResults runs, fail-on-issue.yml catches it ✅ |
CI Status
| Status | Details |
|---|---|
| ⏳ Pending | maui-pr build not yet completed for latest commit |
| AOT integration test failures — same on base branch |
Recommendation: ⚠️ Request Changes
All 6 previous findings resolved ✅. One new critical issue: the in-script error handling needs an elseif branch for the continueOnError=true + useExitCodeForErrors=true case to preserve the old SucceededWithIssues behavior. Without it, test failures are invisible in CI.
|
/azp run maui-pr-uitests, maui-pr-devicetests |
|
Azure Pipelines successfully started running 2 pipeline(s). |
1 similar comment
|
Azure Pipelines successfully started running 2 pipeline(s). |
|
/rebase |
Without -buildVersion, xcodebuild -downloadPlatform iOS downloads the latest available simulator runtime, which may not match the selected Xcode's SDK. On agents with Xcode 26.2, this installs iOS 26.3.1 instead of iOS 26.2, causing actool/ibtool failures: actool: No simulator runtime version from ["22F77"] available to use with iphonesimulator SDK version 23C53 ibtool: iOS 26.2 Platform Not Installed This broke all 17 macOS CI jobs on net11.0 after the SDK bump in PR #34852. Fix: Pin -buildVersion to ${MAJOR}.${MINOR} for Xcode 26.x (where iOS version matches Xcode version). Include -architectureVariant universal for x64 simulator support on Apple Silicon agents. Keep unpinned download for pre-26 Xcode where version mapping differs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…variables Merge the two PowerShell blocks in run-dotnet-preview.yml into one (matching the pattern used by dotnet/android) so that DOTNET_ROOT, DOTNET_MULTILEVEL_LOOKUP, and MSBuildEnableWorkloadResolver are process-scoped only. Only PATH is persisted for subsequent steps. Previously, persisting DOTNET_MULTILEVEL_LOOKUP=0 as a pipeline variable poisoned subsequent pwsh: steps. System PowerShell 7.6 (Homebrew, framework-dependent) needs Microsoft.NETCore.App 10.0.0, but with DOTNET_ROOT pointing to .dotnet/ (only .NET 11 preview) and DOTNET_MULTILEVEL_LOOKUP=0 blocking fallback, pwsh crashed with exit code 150 on all macOS integration test jobs. Also convert a trivial pwsh: step to script: in setup-test-env.yml as a defensive measure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fix echo quoting - Change MAJOR == 26 to MAJOR -ge 26 so Xcode 27+ gets pinned simulator versions instead of silently falling through to unpinned - Add 'Show simulator info' step (xcrun simctl runtime list/match) after simulator install for diagnostics (from rolfbjarne's PR #34979) - Document -architectureVariant universal carry-forward from 26.0 block - Fix echo quoting in setup-test-env.yml for Windows cmd.exe compatibility (cmd.exe prints surrounding quotes, breaking ##vso logging command parsing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
script: runs bash where unquoted ## starts a comment, silently dropping the ##vso logging command. Use bash: with quotes to ensure the ##vso[task.setvariable] is printed correctly on all platforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously the merged step inherited continueOnError: true from the parameter default, which silently swallowed env setup failures (e.g., missing .dotnet/dotnet). Now continueOnError is handled inside the script: env setup failures (PowerShell exceptions, missing binaries) hard-fail the step, while the dotnet command respects the continueOnError parameter for its exit code handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The useExitCodeForErrors block must be guarded by continueOnError=false to preserve the original soft-fail semantics. Without the guard, test failures hard-fail the step even when continueOnError=true (the default), causing downstream PublishTestResults steps to be skipped and test results never published. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ce45cdb to
37341dd
Compare
|
Azure Pipelines successfully started running 2 pipeline(s). |
When continueOnError defaults to true (as in unit/integration test callers), the error handling block was completely skipped, making test failures invisible. Restructure to check commandExitCode first, then branch: - continueOnError=false: hard fail with exit (unchanged) - continueOnError=true + useExitCodeForErrors=true: soft fail — log error and mark task Failed but don't exit, so PublishTestResults still runs and fail-on-issue.yml catches the SucceededWithIssues status - continueOnError=true + useExitCodeForErrors=false: silent continue (unchanged) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Applied the suggested fix for the The error handling now checks
The last row was the broken case — previously the entire error block was skipped when |
|
Azure Pipelines successfully started running 2 pipeline(s). |
jfversluis
left a comment
There was a problem hiding this comment.
This fixes what its supposed to
Problem
All 17 macOS CI jobs on
net11.0are failing after the SDK bump in #34852. The provisioning script runsxcodebuild -downloadPlatform iOSwithout pinning a version, which downloads the latest available simulator runtime (iOS 26.3.1) instead of the one matching the selected Xcode 26.2.The macios SDK
26.2.11588-net11-p3expects iphonesimulator SDK23C53(iOS 26.2), but the installed simulator is iOS 26.3.1 (23D8133). This causes:Evidence
From build 1379721 (
Install Simulator Runtimeslog):Unable to connect to simulator(exit code 70)The Xcode 26.0 case already had this fix — it explicitly pins
-buildVersion 26.0. The bug was only in theelsebranch for Xcode 26.1+.Fix
Pin
-buildVersion ${MAJOR}.${MINOR}for all Xcode 26.x versions (where iOS version matches Xcode version). Include-architectureVariant universalfor x64 simulator support on Apple Silicon agents. Keep unpinned download for pre-26 Xcode where the version mapping differs (Xcode 16.x → iOS 18.x).Reviewed by
Approach validated by 3 independent AI models (Sonnet 4.6, Opus 4.6, Codex 5.3) — all agreed on the fix and flagged the two concerns addressed in the final version:
-architectureVariant universalfor all 26.x (not just 26.0)MAJOR == 26to avoid breaking pre-26 Xcode branches