Repository navigation
Deploy individual modules instead of main.bicep - #11098
Conversation
2c84040 to
bf34199
Compare
bf34199 to
6137c51
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR changes the Azure deployment approach from using a monolithic main.bicep file to deploying individual Bicep modules for each resource. The change aims to improve deployment modularity and resource isolation by removing the centralized aggregation approach.
Key changes:
- Removed
main.bicepaggregation and deploy individual modules directly - Added dependency-based parallel deployment using a resource deployment graph
- Modified test snapshots to provide empty default values for the
userPrincipalIdparameter
Reviewed Changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| Multiple snapshot files | Updated to provide empty default value for userPrincipalId parameter |
| ProvisioningTestHelpers.cs | Enhanced test ARM client provider to support deployment-specific output providers |
| AzureDeployerTests.cs | Updated test data structure to match new individual module deployment approach |
| ResourceDeploymentGraph.cs | New class implementing dependency-based parallel resource deployment |
| BicepProvisioner.cs | Simplified to use timestamped deployment names for all resources |
| AzureEnvironmentResource.cs | Changed from extending AzureBicepResource to Resource, removing main.bicep aggregation |
| AzureDeployingContext.cs | Refactored to deploy individual bicep resources using the deployment graph |
| AzureBicepResource.cs | Added case-insensitive comparison for output dictionaries |
| Extensions files | Added empty default values for userPrincipalId parameters |
Comments suppressed due to low confidence (1)
| { | ||
| return new TestArmDeploymentCollection(_deploymentOutputsProvider); | ||
| } | ||
| return new TestArmDeploymentCollection(_deploymentOutputs!); |
There was a problem hiding this comment.
Multiple instances of the null-forgiving operator (!) are used throughout this file without explicit null validation. Consider using conditional expressions or restructuring the logic to be more defensive about null values.
| return new TestArmDeploymentCollection(_deploymentOutputs!); | |
| if (_deploymentOutputs is null) | |
| { | |
| throw new InvalidOperationException("Deployment outputs are not available."); | |
| } | |
| return new TestArmDeploymentCollection(_deploymentOutputs); |
| } | ||
| #pragma warning restore ASPIREAZURE001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed. | ||
| // Set the location and use timestamped deployment name for all resources | ||
| deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}"; |
There was a problem hiding this comment.
Using DateTimeOffset.Now.ToUnixTimeSeconds() for deployment naming could lead to naming collisions in rapid successive deployments. Consider using a more unique identifier or adding additional entropy.
| deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}"; | |
| deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}-{Guid.NewGuid()}"; |
| } | ||
| #pragma warning restore ASPIREAZURE001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed. | ||
| // Set the location and use timestamped deployment name for all resources | ||
| deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}"; |
There was a problem hiding this comment.
Now that we are not deploying main.bicep, we might not need to disambiguate each resource deployment with a timestamp but I'm keeping it here while we evaluate.
|
|
||
| // This tells azd to avoid creating infrastructure | ||
| var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)); | ||
| var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)) { Value = new BicepValue<string>(string.Empty) }; |
There was a problem hiding this comment.
We need to set a default value for this parameter since we don't expliclty capture it anywhere. An alternative here might be to set a value for it here but I dunno how I feel about treating this as a meaningful parameter when it isn't.
There was a problem hiding this comment.
Yes this is pretty weird. Lets see if we can fix this for the non azd case.
There was a problem hiding this comment.
Why do we need to assign this to make it work?
There was a problem hiding this comment.
When azd generates the main.bicep, it sets the userPrincipalId parameter when passing it through to the module.
We did this two in our generated main.bicep.
Since we're not deploying main.bicep, nobody is setting the parameter.
We could do the alternative above and set it as a known parameter in the BicepProvisioner but I think that leans to much into the hack.
I'm setting a default value to keep ARM from complaining and make it easier to remove this.
FWIW, I think the right thing to do here is figure out a way to bump the version in the manifest and not rely on this workaround for azd.
There was a problem hiding this comment.
When azd generates the main.bicep, it sets the userPrincipalId parameter when passing it through to the module.
If a module requires it, we have to pass it. It's just that we end up requiring it in too many places, but we can fix that.
We could do the alternative above and set it as a known parameter in the BicepProvisioner but I think that leans to much into the hack.
We might be able to lean on this new feature instead https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-deployment.
We can generate deployer().id as the default or something. This does require a newer version of bicep.
FWIW, I think the right thing to do here is figure out a way to bump the version in the manifest and not rely on this workaround for azd.
We can find another way to do this outside of touching the manifest version. That's a HUGE hammer. I like leaning on azure.yaml instead.
There was a problem hiding this comment.
Summarizing some notes from an offline convo.
If a module requires it, we have to pass it. It's just that we end up requiring it in too many places, but we can fix that.
Yes, but the module doesn't actually require it. We (Aspire) are just artificially introducing this parameter as a signal to AZD.
We can find another way to do this outside of touching the manifest version. That's a HUGE hammer. I like leaning on azure.yaml instead.
Sure, I still think that isolating the change to just the module that is artifiacially injecting the parameter is the right thing to do here, especially if we want to work towards removing it. What were trying to say anyway is that it's an optional, artificial parameter and setting a default to empty string achieves that.
| /// Outputs that will be generated by the bicep template. | ||
| /// </summary> | ||
| public Dictionary<string, object?> Outputs { get; } = []; | ||
| public Dictionary<string, object?> Outputs { get; } = new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase); |
There was a problem hiding this comment.
There's a really spooky bug with provisioning ACA environments where the outputs returned in the ARM payload have inconsistent casing (e.g. azurE_CONTAINER_REGISTRY_NAME) which prevents parameters from being filled correctly by the BicepProvisioner. I am fixing this by ignoring casing for output names by default.
There was a problem hiding this comment.
I wonder if we can ask the bicep folks why this happens...
There was a problem hiding this comment.
It's ARM, in azd we handle this by doing toUpper to all outputs names
There was a problem hiding this comment.
Well the semi-good news is that bicep symbolic names are case insensitive. So this change is good either way.
1902f1f to
e960a00
Compare
|
|
||
| // This tells azd to avoid creating infrastructure | ||
| var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)); | ||
| var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)) { Value = new BicepValue<string>(string.Empty) }; |
There was a problem hiding this comment.
Is azd going to be happy with this change? Does it look at the value at all?
There was a problem hiding this comment.
It only looks for a parameter with the name userPrincipalid and an empty string as a value, so I think we're good here.
The reason I had to set a default value within the template is because we no longer deploy main.bicep so we're not settng userPrincipalId: principalId anywhere. It should be fine though since we are not using it.
There was a problem hiding this comment.
We can test it 😄
There was a problem hiding this comment.
azd up works on the current apphost. The manifest we emit has the below which is what azd is expects in this case.
"aca-env": {
"type": "azure.bicep.v0",
"path": "aca-env.module.bicep",
"params": {
"userPrincipalId": ""
}
},
"aas-env": {
"type": "azure.bicep.v0",
"path": "aas-env.module.bicep",
"params": {
"userPrincipalId": ""
}
},
| try | ||
| { | ||
| foreach (var (parameterResource, provisioningParameter) in resource.PublishingContext!.ParameterLookup) | ||
| var deploymentGraph = ResourceDeploymentGraph.Build(bicepResources); |
There was a problem hiding this comment.
The way it works for dotnet run is that a resource that has a BicepOutputReference blocks until that BicepOutputReference is filled with a value. So there is no need to build up a "graph". You just start deploying all resources, and the graph "just works".
See
Can that model be made to work here?
There was a problem hiding this comment.
Maybe? I think we want to extend this model to support representing the image build + push steps. Also, we get to explicitly plug in the publishing activity reporter in this model.
There was a problem hiding this comment.
Yes the implicit graph model works. The downside is that cycles result in a hang😄 (of course this is fixable). Whatever we decide, I'd like to unify run and deploy as much as we can.
There was a problem hiding this comment.
OK the other huge benefit of the runtime graph is concurrency. The first thing I hit testing this PR is that everything is sequential which is a massive PITA.
There was a problem hiding this comment.
The first thing I hit testing this PR is that everything is sequential which is a massive PITA.
Synced about this offline but the we currently deploy Azure resources concurrently in this PR. Practically, that means we start deploying the resources all at once, then we start deploying the role assignments associated with each resource.
There was a problem hiding this comment.
The first thing I hit testing this PR is that everything is sequential which is a massive PITA.
Synced about this offline but the we currently deploy Azure resources concurrently in this PR. Practically, that means we start deploying the resources all at once, then we start deploying the role assignments associated with each resource.
|
|
||
| namespace Aspire.Hosting.Azure; | ||
|
|
||
| internal sealed class ResourceDeploymentGraph |
There was a problem hiding this comment.
What are your thoughts around debugging failed deployments?
Before this PR, there would be one only Azure deployment where to find everything. Azure automatically handles the dependency graph and runs all bicep modules in parallel when possible.
After the PR, will I have as many Azure individual deployments as the number of my resources in the AppHost? Would that become annoying and could probably require some strategy to collect and save the links or names for each one? (like for auditory purposes?)
The new approach might also come with new kind of issues/errros. Like what if the client lost connection in the middle of the deployment. Or how to handle deployment cache to avoid re-deploying things when nothing has changed.
I wonder if, at the end there is a big win versus what people get from delegating ARM service the full deployment.
There was a problem hiding this comment.
After the PR, will I have as many Azure individual deployments as the number of my resources in the AppHost? Would that become annoying and could probably require some strategy to collect and save the links or names for each one? (like for auditory purposes?)
Yep, you'd get a separate deployment for each resource. However, because we deploy the resource group independently they are all resource group-scoped deployments instead of subscription scoped deployments so the noise is a little less intense.
Currently, we create a new deployment name for each resource and deployment but I can see us removing this. The only thing we'd lose is deployment history in the portal but that might be fine?
The new approach might also come with new kind of issues/errros. Like what if the client lost connection in the middle of the deployment. Or how to handle deployment cache to avoid re-deploying things when nothing has changed.
I believe the desired state/deployment cache rules still apply for single resource deployments as well.
I wonder if, at the end there is a big win versus what people get from delegating ARM service the full deployment.
Yep, there's some heeming and hawing about this. So far the pain benefits of this approach are:
- The ability to inject other steps between resource deployments
- The ability to model the state of each deploymenet within the AppHost instead of relegating it to ARM
* Deploy individual modules instead of main.bicep * Bring back generating main.bicep and fix userPrincipalId * Update deployment data mocks for single resources * Update snapshots and tests * Support deploying infrastructure resources in parallel * Don't snapshot resource lists * Remove OutputPath dependency from BicepProvisioner * Fix test * ActiveIssue on flaky test
Contributes to #10448.