Skip to content

Rejected transitive central dependencies should be removed from the restore graph - #3374

Merged
cristinamanum merged 18 commits into
devfrom
dev-cmanu-9483
Jun 30, 2020
Merged

cristinamanum merged 18 commits into
devfrom
dev-cmanu-9483

Conversation

@cristinamanum

@cristinamanum cristinamanum commented May 1, 2020 •

Copy link
Copy Markdown
Contributor

Bug

Fixes issue: Rejected transitive central dependencies should be removed from the restore graph.

A visualization of the problem:

Project
          -> PackageA 1.0.0 -> PackageB 1.0.0 -> CentralPackageC 1.0.0 ->...
          -> PackageD 2.0.0 -> PackageB 2.0.0

During the graph resolution PackageB 1.0.0 becomes Rejected and will be dropped from the graph in the favor of PackageB 2.0.0. In this context the CentralPackageC 1.0.0 should not be in the graph anymore. The flatten graph should include only PackageA 1.0.0, PackageD 2.0.0, PackageB 2.0.0

Without this fix the flatten graph is:
PackageA 1.0.0, PackageD 2.0.0, PackageB 2.0.0, CentralPackageC 1.0.0

Fix

Details:
The nodes added as central dependency nodes keep reference of their parents.

In the case of opted in central package versions projects, the restore graph is analyzed :

  1. The central transitive dependencies will become ambiguous if all their parents are either Ambiguous or Disputed.
  2. The Central Transitive dependencies are rejected if their parents are rejected.

Testing/Validation

Tests Added: Yes
Reason for not adding tests:
Validation: Unit/functional test and manual testing

@cristinamanum cristinamanum changed the title Initial changes to deal with rejected central transitive nodes. Rejected transitive central dependencies should be removed from the restore graph [Draft] May 1, 2020
@cristinamanum
cristinamanum requested review from nkolev92 and zivkan May 1, 2020 19:39
@cristinamanum
cristinamanum marked this pull request as ready for review May 20, 2020 21:45
@cristinamanum cristinamanum changed the title Rejected transitive central dependencies should be removed from the restore graph [Draft] Rejected transitive central dependencies should be removed from the restore graph May 21, 2020
@cristinamanum
cristinamanum requested a review from zivkan June 3, 2020 16:42
};
}

private static bool TryResolveConflictsForCPVM<TItem>(this GraphNode<TItem> root, List<VersionConflictResult<TItem>> versionConflicts)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing a diff between TryResolveConflictsForCPVM and TryResolveConflicts, there are only 2 differences:

  1. ForCPVM version calls an overload of ForEach with an extra skipNode parameter. However, the ForEach overload with skipNode calls the overload with skipNode with a null value. Therefore, if this method had the skipNode func as a method parameter, then both the CPVM and non-CPVM codepaths could use the same codepath.

  2. CPVM version calls RejectCentralTransitiveBecauseOfRejectedParents. RejectCentralTransitiveBecauseOfRejectedParents looks like a no-op for non-CPVM graphs. Alternatively we chould guard the method call in an if statement.

It looks to me like we can get rid of this new method, which I think is important because it implements business logic, (not just trivial code), so this type of code duplication is real technical debt that can lead to bugs if one of the two implementations change in the future.

@cristinamanum cristinamanum Jun 3, 2020 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would let the two methods different. In my opinion it is easier to follow the logic in this way. It will also reduce the regression risks to existent path.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long term, I'm with the @zivkan.
While it does make it easier to follow given the context we have today, have 2 different code paths with duplication would just amplify the confusion for a new reader.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@cristinamanum
cristinamanum requested a review from zivkan June 3, 2020 22:51
};
}

private static bool TryResolveConflictsForCPVM<TItem>(this GraphNode<TItem> root, List<VersionConflictResult<TItem>> versionConflicts)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long term, I'm with the @zivkan.
While it does make it easier to follow given the context we have today, have 2 different code paths with duplication would just amplify the confusion for a new reader.

@davidfowl

Copy link
Copy Markdown
Member

Do we have any performance numbers? I see a couple of things that stand out to me but it would be nice to have some microbenchmarks for what we consider best and worse case so we can compare before and after this change.

@cristinamanum

Copy link
Copy Markdown
Contributor Author

Do we have any performance numbers? I see a couple of things that stand out to me but it would be nice to have some microbenchmarks for what we consider best and worse case so we can compare before and after this change.

Talked with @nkolev92 offline regarding the comment

Wouldn't it be more efficient to add all the discovered conflicts as top level and then call CreateGraphNode again? Now you run phase 1, you detect that B & C need bumping. You currently add them 1 by 1.
#3374 (comment)

The code is behaving as expected.

@davidfowl please let me know if there are additional suggestions that I need to look into?

@cristinamanum
cristinamanum requested a review from nkolev92 June 11, 2020 16:55

@zivkan zivkan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll need to rebase your branch onto the latest dev to get the build to work. MicroBuild's new CI agents have an issue that doesn't work with the VS account management assembly, so we had to skip those tests for the time being.

/// <summary>
/// Used in case that a node is removed from its outernode and needs to keep reference of its parents.
/// </summary>
internal BlockingCollection<GraphNode<TItem>> ParentNodes { get; }

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using a blocking collection? Is it solely because of IsAddingCompleted?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it was because the IsAddingCompleted. However this was an extra and probably not necessary caution because the iterations of the collections is executed after all the parents are added. Running performance tests iterating over BlockingCollection is more expensive so changed to IList.

if (!n.ParentNodes.Any(p => !(tracker.IsDisputed(p.Item) || tracker.IsAmbiguous(p.Item))))
{
tracker.MarkAmbiguous(n.Item);
n.ForEach(x => tracker.MarkAmbiguous(x.Item));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to do this explicit walk of child nodes here? Can you do this in a single pass without the nested ForEach by moving the predicate?

@cristinamanum cristinamanum Jun 25, 2020 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a single pass over each of the central transitive nodes. I added a predicate to skip any nodes that are already marked as Ambiguous, I am not sure if I addressed your concerns though.

});
}

private static void RejectCentralTransitiveBecauseOfRejectedParents<TItem>(this GraphNode<TItem> root)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really expensive to run on each node both from a big O perspective and number of allocations perspective. That's why I was asking about performance tests. The ASP.NET Core repo has > 100 central dependencies, does this perform well at that scale?

@cristinamanum cristinamanum Jun 25, 2020 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the big O can be large depending on the graph structure. I did run benchmarks for rejects/acceptances scenarios. The results are below (graph used has 2,047 nodes and a subset of these are central transitive). I will still keep thinking of solutions to reduce it.
The total central dependencies count is important, for the CPVM scenarios the count of the subset that becomes transitive dependencies is more important.

Name Time milliseconds
2_10_2_AnalyseGraphWith4AcceptedCentralTransitiveDeps 2
2_10_2_AnalyseGraphWith4RejectedCentralTransitiveDependencies 3.56
2_10_2_AnalyseGraphWithoutCentralTransitiveDepsAsync_HalfGraphRejected 2.66
2_10_4_AnalyseGraphWith16AcceptedCentralTransitiveDeps 2.61
2_10_4_AnalyseGraphWith16RejectedCentralTransitiveDependencies 4.06
2_10_4_AnalyseGraphWithoutCentralTransitiveDepsAsync_HalfGraphRejected 2.82
2_10_6_AnalyseGraphWith64AcceptedCentralTransitiveDeps 3.09
2_10_6_AnalyseGraphWith64RejectedCentralTransitiveDependencies 4.31
2_10_6_AnalyseGraphWithoutCentralTransitiveDepsAsync_HalfGraphRejected 3.09

@cristinamanum
cristinamanum merged commit d0d8e4d into dev Jun 30, 2020
@cristinamanum
cristinamanum deleted the dev-cmanu-9483 branch June 30, 2020 17:28
@nkolev92

Copy link
Copy Markdown
Member

I would've wanted to review this first before merging.

There had been many changes since I was last requested for review, so I wasn't sure when to take a look.

The code now looks significantly different from when it was approved.

@davidfowl

Copy link
Copy Markdown
Member

@cristinamanum please file an issue for the potential performance problems with this

@cristinamanum

Copy link
Copy Markdown
Contributor Author

please file an issue for the potential performance problems with this

Added NuGet/Home#9740

@cristinamanum

Copy link
Copy Markdown
Contributor Author

I would've wanted to review this first before merging.

There had been many changes since I was last requested for review, so I wasn't sure when to take a look.

The code now looks significantly different from when it was approved

The changes performed were around trying to get the best performance that I could based on benchmarks data. Please let me know additional feedback and I will be happy to address it.

cristinamanum added a commit that referenced this pull request Jul 1, 2020
nkolev92 pushed a commit that referenced this pull request Jul 1, 2020
@cristinamanum
cristinamanum restored the dev-cmanu-9483 branch July 1, 2020 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants