Repository navigation
Undo revert of Arcade update to ingest Workload Manifest task #71150
Description
Basically, undo #71110 (but don't undo the SDK update), and fix up the publish task as described below (copy/pasted from the PR):
Runtime will have to take an arcade update, a llvm-project update, and Jacques' change next month, plus a fix to the publish stage invoked here: https://github.com/dotnet/runtime/blob/cd19e8bf35d18cef6f30d896313cbeb1be2a06dd/eng/pipelines/official/jobs/prepare-signed-artifacts.yml#L48-L57. The problem seems to be that you're trying to upload all blobs to `blobArtifacts`, but there are now .msi's with duplicate names. From @joeloff:
> [7:52 PM] Jacques Eloff
it organizes builds by feature band for VS insertions, so MSIs can be duplicated in multiple folders. Copying them the same location won't work.
> because in staging, each sets of files for each feature band is published to a separate VSDROP container and there can be overlap, but not all blobbies have the same files.
Wanted to make sure this was on your radar. The failure happens during the AzDO artifact upload task that happens at the end of that step:
> Async Command Start: Upload Artifact
> ...
> Fail to upload 'D:\a\_work\1\s\artifacts\PackageDownload\IntermediateArtifacts\workloads-vs\Insertion\6.0.300\Microsoft.AOT.win-x64.Cross.android-arm.6.0.7.msi' due to 'Blob is incomplete (missing block). Blob: af95b7c0d1f1ec11b656501ac504b8de, Expected Offset: 0, Actual Offset: 4194304'.
The root cause of this was the change to the workload manifest tasks in Arcade: dotnet/arcade#8645
This requires reaction in runtime's usage of those tasks:
runtime/src/workloads/workloads.csproj
Lines 113 to 133 in 6212eab
We tried to react with #71046, but that caused us to try to publish multiple .msi's with the same name to the same place. Basically we need to publish the output of the new task & preserve its output folder structure, rather than flattening.
Activity
ericstj commentedon Jun 22, 2022
@directhex @steveisok can you prioritize a change to workloads.csproj to fix this and help get Arcade flowing again in 6.0?
joeloff commentedon Jun 22, 2022
Some details. The workload outputs will be organized by feature band to facilitate publishing the specific content for the VS versions being targeted. There's some automation tasks that we have in staging that we haven't been able to use.
artifacts\VSSetup\Debug\Insertion\6.0.100
artifacts\VSSetup\Debug\Insertion\6.0.200
artifacts\VSSetup\Debug\Insertion\6.0.300
Because workload packs can be the same, it's likely there will be overlap in these folders and it seems that publishing was pushing the content to the same blob container
directhex commentedon Jun 30, 2022
Hi @wtgodbe, do logs still exist for one of the builds which failed due to the Arcade bump?
wtgodbe commentedon Jun 30, 2022
https://dev.azure.com/dnceng/internal/_build/results?buildId=1837162&view=logs&j=138e5e17-b2cd-527a-2da4-df51693fdead&t=7c441ab8-f67c-56a1-1eef-407c16f44b55&s=fbeceea2-8819-56cc-d4aa-12918796648b
directhex commentedon Jun 30, 2022
OK, that binlog has revealed a lot
The
publishsubset runs thoughsrc/installer/prepare-artifacts.proj. One thing happening in that file is here;runtime/src/installer/prepare-artifacts.proj
Lines 121 to 124 in 9aec9ad
@(WorkloadsVSInsertionFile)contains the paths of all files generated by the workload generation task which we want to keep, but becauseSdkBandVersionis static (fromVersions.props), that's the specific place where all three versions of the workload files get an identical targetRelativeBlobPathof6.0.300. With any luck, fixing this up to derive from the source path will be enough (maybe settingPublishFlatContainertofalseis also needed, will see)directhex commentedon Jul 1, 2022
This is increasingly looking like an annoying problem to resolve.
PushToAzureDevOpsArtifacts, part of Arcade, is apparently where things are going wrong. If the Items have aPublishFlatContainer=falseproperty attached, then directory structure is not flattened... but the task will fail for non-nupkg files (e.g. json files). IfPublishFlatContainer=truethen we run into the observed behaviour withSystem.InvalidOperationException: Blob is incomplete (missing block).and non-zero offsets.So I need to pull Arcade apart to try and fix it (and really need to set up a minimal repro case so I'm not waiting hours for every attempt)
directhex commentedon Jul 5, 2022
I can confirm my analysis from a simple repro case. https://github.com/directhex/arcade-repro runs on https://dev.azure.com/joshield/testing/_build?definitionId=1&_a=summary and behaves in exactly the same way as runtime.git:
PublishFlatContainer=falseonly supports NuPkg files, andRelativeBlobPathis thrown away during the upload.PublishToAzureDevOpsArtifactsjust doesn't support folders in the general case.For small files (smaller than the upload chunk size), the last uploaded file simply overwrites older files of the same name. For files larger than the chunk size, you get the
Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Bad Requestwhich is largely typified byBlob is incomplete (missing block).Should I open an issue against Arcade for this?
joeloff commentedon Jul 5, 2022
Yeah,. might be worth to pull them in and see if there's a good solution. Right now we have duplicated packs, but the idea was always that workloads for different feature bands can diverge and we want to upload clean drops for VS for each feature band.
directhex commentedon Aug 30, 2022
@joeloff what's the status of the issue here? Are we still producing files with duplicate names in workloads? Or have other discussions surrounding multi-targeting of workloads affected the underlying intent behind this PR?
joeloff commentedon Aug 30, 2022
No, we should close this. Let me link the PRs for this.
joeloff commentedon Aug 30, 2022
Fixed with #73981