Skip to content

Deploy individual modules instead of main.bicep - #11098

Merged
Safia Abdalla (captainsafia) merged 10 commits into
mainfrom
safia/deploy-split
Aug 28, 2025
Merged

Safia Abdalla (captainsafia) merged 10 commits into
mainfrom
safia/deploy-split

Conversation

@captainsafia

@captainsafia Safia Abdalla (captainsafia) commented Aug 26, 2025 •

Copy link
Copy Markdown
Contributor

Contributes to #10448.

20250827_141958

@github-actions github-actions Bot added the area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication label Aug 26, 2025
@captainsafia
Safia Abdalla (captainsafia) force-pushed the safia/deploy-split branch 2 times, most recently from 2c84040 to bf34199 Compare August 27, 2025 02:46
@captainsafia
Safia Abdalla (captainsafia) marked this pull request as ready for review August 27, 2025 21:27
Copilot AI review requested due to automatic review settings August 27, 2025 21:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR changes the Azure deployment approach from using a monolithic main.bicep file to deploying individual Bicep modules for each resource. The change aims to improve deployment modularity and resource isolation by removing the centralized aggregation approach.

Key changes:

  • Removed main.bicep aggregation and deploy individual modules directly
  • Added dependency-based parallel deployment using a resource deployment graph
  • Modified test snapshots to provide empty default values for the userPrincipalId parameter

Reviewed Changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
Multiple snapshot files Updated to provide empty default value for userPrincipalId parameter
ProvisioningTestHelpers.cs Enhanced test ARM client provider to support deployment-specific output providers
AzureDeployerTests.cs Updated test data structure to match new individual module deployment approach
ResourceDeploymentGraph.cs New class implementing dependency-based parallel resource deployment
BicepProvisioner.cs Simplified to use timestamped deployment names for all resources
AzureEnvironmentResource.cs Changed from extending AzureBicepResource to Resource, removing main.bicep aggregation
AzureDeployingContext.cs Refactored to deploy individual bicep resources using the deployment graph
AzureBicepResource.cs Added case-insensitive comparison for output dictionaries
Extensions files Added empty default values for userPrincipalId parameters
Comments suppressed due to low confidence (1)

{
return new TestArmDeploymentCollection(_deploymentOutputsProvider);
}
return new TestArmDeploymentCollection(_deploymentOutputs!);

Copilot AI Aug 27, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multiple instances of the null-forgiving operator (!) are used throughout this file without explicit null validation. Consider using conditional expressions or restructuring the logic to be more defensive about null values.

Suggested change
return new TestArmDeploymentCollection(_deploymentOutputs!);
if (_deploymentOutputs is null)
{
throw new InvalidOperationException("Deployment outputs are not available.");
}
return new TestArmDeploymentCollection(_deploymentOutputs);

Copilot uses AI. Check for mistakes.
}
#pragma warning restore ASPIREAZURE001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
// Set the location and use timestamped deployment name for all resources
deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}";

Copilot AI Aug 27, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using DateTimeOffset.Now.ToUnixTimeSeconds() for deployment naming could lead to naming collisions in rapid successive deployments. Consider using a more unique identifier or adding additional entropy.

Suggested change
deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}";
deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}-{Guid.NewGuid()}";

Copilot uses AI. Check for mistakes.
}
#pragma warning restore ASPIREAZURE001 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
// Set the location and use timestamped deployment name for all resources
deploymentName = $"{resource.Name}-{DateTimeOffset.Now.ToUnixTimeSeconds()}";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are not deploying main.bicep, we might not need to disambiguate each resource deployment with a timestamp but I'm keeping it here while we evaluate.


// This tells azd to avoid creating infrastructure
var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string));
var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)) { Value = new BicepValue<string>(string.Empty) };

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to set a default value for this parameter since we don't expliclty capture it anywhere. An alternative here might be to set a value for it here but I dunno how I feel about treating this as a meaningful parameter when it isn't.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is pretty weird. Lets see if we can fix this for the non azd case.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to assign this to make it work?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When azd generates the main.bicep, it sets the userPrincipalId parameter when passing it through to the module.

We did this two in our generated main.bicep.

Since we're not deploying main.bicep, nobody is setting the parameter.

We could do the alternative above and set it as a known parameter in the BicepProvisioner but I think that leans to much into the hack.

I'm setting a default value to keep ARM from complaining and make it easier to remove this.

FWIW, I think the right thing to do here is figure out a way to bump the version in the manifest and not rely on this workaround for azd.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When azd generates the main.bicep, it sets the userPrincipalId parameter when passing it through to the module.

If a module requires it, we have to pass it. It's just that we end up requiring it in too many places, but we can fix that.

We could do the alternative above and set it as a known parameter in the BicepProvisioner but I think that leans to much into the hack.

We might be able to lean on this new feature instead https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-deployment.

We can generate deployer().id as the default or something. This does require a newer version of bicep.

FWIW, I think the right thing to do here is figure out a way to bump the version in the manifest and not rely on this workaround for azd.

We can find another way to do this outside of touching the manifest version. That's a HUGE hammer. I like leaning on azure.yaml instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summarizing some notes from an offline convo.

If a module requires it, we have to pass it. It's just that we end up requiring it in too many places, but we can fix that.

Yes, but the module doesn't actually require it. We (Aspire) are just artificially introducing this parameter as a signal to AZD.

We can find another way to do this outside of touching the manifest version. That's a HUGE hammer. I like leaning on azure.yaml instead.

Sure, I still think that isolating the change to just the module that is artifiacially injecting the parameter is the right thing to do here, especially if we want to work towards removing it. What were trying to say anyway is that it's an optional, artificial parameter and setting a default to empty string achieves that.

/// Outputs that will be generated by the bicep template.
/// </summary>
public Dictionary<string, object?> Outputs { get; } = [];
public Dictionary<string, object?> Outputs { get; } = new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a really spooky bug with provisioning ACA environments where the outputs returned in the ARM payload have inconsistent casing (e.g. azurE_CONTAINER_REGISTRY_NAME) which prevents parameters from being filled correctly by the BicepProvisioner. I am fixing this by ignoring casing for output names by default.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can ask the bicep folks why this happens...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ARM, in azd we handle this by doing toUpper to all outputs names

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well the semi-good news is that bicep symbolic names are case insensitive. So this change is good either way.

https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/resource-declaration?utm_source=chatgpt.com&tabs=azure-powershell#define-resources


// This tells azd to avoid creating infrastructure
var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string));
var userPrincipalId = new ProvisioningParameter(AzureBicepResource.KnownParameters.UserPrincipalId, typeof(string)) { Value = new BicepValue<string>(string.Empty) };

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is azd going to be happy with this change? Does it look at the value at all?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It only looks for a parameter with the name userPrincipalid and an empty string as a value, so I think we're good here.

The reason I had to set a default value within the template is because we no longer deploy main.bicep so we're not settng userPrincipalId: principalId anywhere. It should be fine though since we are not using it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can test it 😄

cc Victor Vazquez (@vhvb1989)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azd up works on the current apphost. The manifest we emit has the below which is what azd is expects in this case.

"aca-env": {
  "type": "azure.bicep.v0",
  "path": "aca-env.module.bicep",
  "params": {
    "userPrincipalId": ""
  }
},
"aas-env": {
  "type": "azure.bicep.v0",
  "path": "aas-env.module.bicep",
  "params": {
    "userPrincipalId": ""
  }
},

try
{
foreach (var (parameterResource, provisioningParameter) in resource.PublishingContext!.ParameterLookup)
var deploymentGraph = ResourceDeploymentGraph.Build(bicepResources);

@eerhardt Eric Erhardt (eerhardt) Aug 27, 2025 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way it works for dotnet run is that a resource that has a BicepOutputReference blocks until that BicepOutputReference is filled with a value. So there is no need to build up a "graph". You just start deploying all resources, and the graph "just works".

See

https://github.com/dotnet/aspire/blob/f7f1efc859c475094428b9181ffc15fbc08c1446/src/Aspire.Hosting.Azure/AzureBicepResource.cs#L383-L386

Can that model be made to work here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe? I think we want to extend this model to support representing the image build + push steps. Also, we get to explicitly plug in the publishing activity reporter in this model.

@davidfowl David Fowler (davidfowl) Aug 28, 2025 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the implicit graph model works. The downside is that cycles result in a hang😄 (of course this is fixable). Whatever we decide, I'd like to unify run and deploy as much as we can.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK the other huge benefit of the runtime graph is concurrency. The first thing I hit testing this PR is that everything is sequential which is a massive PITA.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first thing I hit testing this PR is that everything is sequential which is a massive PITA.

Synced about this offline but the we currently deploy Azure resources concurrently in this PR. Practically, that means we start deploying the resources all at once, then we start deploying the role assignments associated with each resource.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first thing I hit testing this PR is that everything is sequential which is a massive PITA.

Synced about this offline but the we currently deploy Azure resources concurrently in this PR. Practically, that means we start deploying the resources all at once, then we start deploying the role assignments associated with each resource.


namespace Aspire.Hosting.Azure;

internal sealed class ResourceDeploymentGraph

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are your thoughts around debugging failed deployments?

Before this PR, there would be one only Azure deployment where to find everything. Azure automatically handles the dependency graph and runs all bicep modules in parallel when possible.

After the PR, will I have as many Azure individual deployments as the number of my resources in the AppHost? Would that become annoying and could probably require some strategy to collect and save the links or names for each one? (like for auditory purposes?)

The new approach might also come with new kind of issues/errros. Like what if the client lost connection in the middle of the deployment. Or how to handle deployment cache to avoid re-deploying things when nothing has changed.

I wonder if, at the end there is a big win versus what people get from delegating ARM service the full deployment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After the PR, will I have as many Azure individual deployments as the number of my resources in the AppHost? Would that become annoying and could probably require some strategy to collect and save the links or names for each one? (like for auditory purposes?)

Yep, you'd get a separate deployment for each resource. However, because we deploy the resource group independently they are all resource group-scoped deployments instead of subscription scoped deployments so the noise is a little less intense.

Currently, we create a new deployment name for each resource and deployment but I can see us removing this. The only thing we'd lose is deployment history in the portal but that might be fine?

The new approach might also come with new kind of issues/errros. Like what if the client lost connection in the middle of the deployment. Or how to handle deployment cache to avoid re-deploying things when nothing has changed.

I believe the desired state/deployment cache rules still apply for single resource deployments as well.

I wonder if, at the end there is a big win versus what people get from delegating ARM service the full deployment.

Yep, there's some heeming and hawing about this. So far the pain benefits of this approach are:

  • The ability to inject other steps between resource deployments
  • The ability to model the state of each deploymenet within the AppHost instead of relegating it to ARM

@captainsafia
Safia Abdalla (captainsafia) deleted the safia/deploy-split branch August 28, 2025 19:45
@dotnet-policy-service dotnet-policy-service Bot added this to the 9.5 milestone Aug 28, 2025
Copilot AI pushed a commit that referenced this pull request Sep 3, 2025
* Deploy individual modules instead of main.bicep

* Bring back generating main.bicep and fix userPrincipalId

* Update deployment data mocks for single resources

* Update snapshots and tests

* Support deploying infrastructure resources in parallel

* Don't snapshot resource lists

* Remove OutputPath dependency from BicepProvisioner

* Fix test

* ActiveIssue on flaky test
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 28, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants