Repository navigation
Many Android JIT tests failing in CI #44306
Description
Configuration: Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Open
@akoeplinger @steveisok is this known?
Runfo Tracking Issue: Many Android JIT tests failing in CI
| Build | Definition | Kind | Run Name | Console | Core Dump | Test Results | Run Client |
|---|---|---|---|---|---|---|---|
| 956620 | runtime | PR 47182 | net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open | console.log | runclient.py | ||
| 945073 | runtime | PR 46810 | net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open | console.log | runclient.py | ||
| 944893 | runtime | PR 46810 | net6.0-Android-Release-arm64-Mono_Release-Windows.10.Amd64.Android.Open | console.log | runclient.py |
Build Result Summary
| Day Hit Count | Week Hit Count | Month Hit Count |
|---|---|---|
| 0 | 0 | 0 |
Activity
Dotnet-GitSync-Bot commentedon Nov 5, 2020
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
[-]Many Android tests failing in rolling build CI[/-][+]Many Android runtime tests failing in rolling build CI[/+][-]Many Android runtime tests failing in rolling build CI[/-][+]Many Android JIT tests failing in CI[/+]ViktorHofer commentedon Nov 5, 2020
cc @SamMonoRT @vargaz
ViktorHofer commentedon Nov 5, 2020
@danmosemsft the bot isn't mentioning anyone, intentional?
steveisok commentedon Nov 5, 2020
So, https://dev.azure.com/dnceng/public/_build/results?buildId=873257&view=ms.vss-test-web.build-test-results-tab is a result of an apk installation failure. That impacted 50+ tests on two runs from #38457
It looks like work items completed on the machine that failed afterwards, so my guess is that adb is in a bad state and may be corrected upon the work item finishing. We can look into making this more resilient.
The other issues linked in by runfo appear to be different. My guess on some would adb corruption occurring at different points in time.
danmoseley commentedon Nov 5, 2020
The owner is @vargaz -- @vargaz do you want notifications for this area label?
akoeplinger commentedon Nov 5, 2020
@ViktorHofer all of the failures from #43740 are known, we're iterating on replacing OpenSSL on Android there.
ViktorHofer commentedon Nov 11, 2020
@steveisok what do you think would be a reasonable follow-up action? The day hit count is concerningly high. Assuming that disabling the Android leg in CI is a no go, should we move the job into a different pipeline that doesn't fail the build: #44031?
cc @safern @jkoritzinsky for opinions
fanyang-mono commentedon Nov 11, 2020
I wonder why this only happened to runtime tests? Theoretically, library tests should be affected as well.
steveisok commentedon Nov 11, 2020
I'm trying to sort through the recent data first and that may help what a good follow up action will be.
8 remaining items
Ingest dotnet/runtime master (#385)
ViktorHofer commentedon Nov 30, 2020
I see tons of Android failures in many PRs today :(
safern commentedon Dec 1, 2020
Note that I just merged my staging pipeline PR which moved the Android tests that use the emulator.
fanyang-mono commentedon Apr 19, 2021
Runtime tests have been running reliably on Android x64 for quite a while now. I wonder if it could be moved to a regular pipeline? Lately, there are real issues weren't fixed by the introduced PR, because the staging lane checks are always green.
ViktorHofer commentedon Apr 19, 2021
"Reliably" isn't measurable. If the android runtime tests pass rate numbers in the last 7-14 days have been above 90% then I agree that they should be moved into the main pipeline.
fanyang-mono commentedon Apr 19, 2021
Where could I get that data?
ViktorHofer commentedon Apr 19, 2021
AzDO REST APIs or Kusto. You will likely need to write a custom query to filter on just the android runtime tests.
ViktorHofer commentedon Apr 19, 2021
maybe @steveisok already has a query for that?
SamMonoRT commentedon Jun 24, 2021
@fanyang-mono : I think we have most of these tests passing after your recent PRs, and the ones which are failing have individual issues logged. Do we still need this open ?
fanyang-mono commentedon Jun 24, 2021
Lane
Mono Android x64 Release @ Ubuntu.1804.Amd64.Android.Openhas been stable and moved back to regular pipeline. Closing this issue.