Repository navigation
Using Apphost for post netcoreapp2.0 standalone activation - #978
Conversation
| namespace Microsoft.NET.Build.Tasks | ||
| { | ||
| /// <summary> | ||
| /// Embeds the App Name into the Nuetered host |
There was a problem hiding this comment.
Nuetered should probably not be used here.
There was a problem hiding this comment.
Definitely not. (And not just because it's misspelled)
| var _modifiedAppHostPath = Path.Combine(destinationDirectory, $"{appbaseName}{hostExtension}"); | ||
| _outputAppHostPath= new TaskItem(_modifiedAppHostPath); | ||
|
|
||
| if ( File.Exists(_modifiedAppHostPath)) |
There was a problem hiding this comment.
?? Shouldn't we overwrite it if it already exists? What if it doesn't contain the correct information?
There was a problem hiding this comment.
The info inside is the name of the app, so if we have placed a file here- there is not much to do.
Do you think a comment will make it more clearer
|
|
||
| var appbaseName = Path.GetFileNameWithoutExtension(AppName); | ||
| var destinationDirectory = Path.GetFullPath(AppHostDestinationDirectoryPath); | ||
| var _modifiedAppHostPath = Path.Combine(destinationDirectory, $"{appbaseName}{hostExtension}"); |
There was a problem hiding this comment.
local variables should not start with _. Private fields should.
| --> | ||
|
|
||
| <Target Name="RenamePublishedHost" | ||
| DependsOnTargets="_ComputeNETCoreBuildOutputFiles" |
There was a problem hiding this comment.
Why? That target is for "build", this is for "publish".
There was a problem hiding this comment.
publish pipes in from build, the notion is largely artificial
| </ItemGroup> | ||
|
|
||
| <ItemGroup Condition=" '$(DisableImplicitFrameworkReferences)' != 'true' and '$(TargetFrameworkIdentifier)' == '.NETCoreApp' and !$(RuntimeFrameworkVersion.StartsWith('1.'))"> | ||
| <PackageReference Include="Microsoft.NETCore.DotNetAppHost" Version="$(RuntimeFrameworkVersion)" IsImplicitlyDefined="true" /> |
There was a problem hiding this comment.
Won't this result in always referencing this package? Even if I'm a portable app?
There was a problem hiding this comment.
That is the meta package, it will not resolved to any assets
There was a problem hiding this comment.
It's going to bleed in to pack, solution explorer, etc. etc. cc @davidfowl
There was a problem hiding this comment.
What is the point of having the Microsoft.NETCore.DotNetHost package anymore? We will never use the dotnet executable from it, since standalone apps always use DotNetAppHost.
So instead of doing it this way, why don't we just put the apphost executable into the DotNetHost package?
Alternatively, change Microsoft.NETCore.App to reference Microsoft.NETCore.DotNetAppHost and remove the reference to Microsoft.NETCore.DotNetHost.
There was a problem hiding this comment.
https://github.com/dotnet/core-setup/issues/1722
but that does not affect the change here in SDK
It does affect the change here in the SDK. If we configure the packages efficiently, the tooling doesn't need all this logic in it. See the issue above. In this change we are doing things like
- Delete the
dotnet.exefile from the published files. - First look for
apphost.exe, if that isn't found fall back todotnet.exe.
The tooling logic can be a lot simpler.
There was a problem hiding this comment.
I agree it can be made simpler once we have figured the correct changes for Core-Setup. However, that should not block enabling this feature. @ramarag please context to this discussion in the issue above so that, once the feature is enabled, you should visit cleaning up Core-Setup, followed by the SDK.
There was a problem hiding this comment.
@eerhardt i have already done what you are suggesting.
@gkhanna79 the latest M.N.App package indeed refers the Apphost package, but the one sdk currently refers does not, will update it to the latest version and remove this reference
There was a problem hiding this comment.
Actually spoke too soon, i will have to update M.N.App to get apphost.exe in it
| <ResolvedFileToRemove Include ="%(ResolvedFileToPublish.Identity)" Condition="'%(ResolvedFileToPublish.RelativePath)' == '$(_DotNetHostExecutableName)'"/> | ||
| <ResolvedFileToPublish Remove ="%(ResolvedFileToRemove.Identity)"/> | ||
|
|
||
| <ResolvedFileToAdd Include="%(NativeAppHostNETCore.Identity)"> |
There was a problem hiding this comment.
Why use 2 items here? Why not just Include this into <ResolvedFileToPublish>?
There was a problem hiding this comment.
Because i need to set the metadata on the item
There was a problem hiding this comment.
You can set metadata on the item even without using a separate item type here:
<ResolvedFileToPublish Include="%(NativeAppHostNETCore.Identity)">
<RelativePath>$(AssemblyName)%(Extension)</RelativePath>
</ResolvedFileToPublish>
Works and is simpler and more performant.
There was a problem hiding this comment.
This sets the metadata on all ResolvedFileToPublish, not what we actually want
There was a problem hiding this comment.
No it doesn't. Just like:
<Content Include="File1">
</Content>
<Content Include="File2">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</Content>
In that scenario, how many files get copied to the output directory? Only File2. Adding a new Item and setting metadata on the item doesn't set the metadata on all the existing files in that item.
| <data name="FileNameIsTooLong" xml:space="preserve"> | ||
| <value>Given File Name '{0}' is longer than 1024 bytes</value> | ||
| </data> | ||
| <data name="HostIsNotNuetered" xml:space="preserve"> |
There was a problem hiding this comment.
Again, please don't use Nuetered.
| var rid = EnvironmentInfo.GetCompatibleRid(targetFramework); | ||
|
|
||
|
|
||
| TestProject referencerProject = new TestProject() |
| IsSdkProject = true, | ||
| TargetFrameworks = targetFramework, | ||
| RuntimeFrameworkVersion = RepoInfo.NetCoreApp20Version, | ||
| // Need to use a self-contained app for now because we don't use a CLI that has a "2.0" shared framework |
There was a problem hiding this comment.
this comment doesn't belong. We use a self-contained app because thatis what we are testing.
| <EmbedAppNameInHost AppHostSourcePath="@(NativeRestoredAppHostNETCore)" | ||
| AppHostDestinationDirectoryPath="$(AppHostDestinationDirectoryPath)" | ||
| AppName="$(AssemblyName).dll" | ||
| StringToReplace="foobar" |
There was a problem hiding this comment.
Foobar is among terms that are not to be used in products. Also, the blind search and replace strategy carries risk that the mere 5 bytes occur elsewhere in the exe with completely different meaning.
There was a problem hiding this comment.
blame @schellap for this, does it make more sense to move this string into C# or just hard code the sha value ?
|
|
||
| <EmbedAppNameInHost AppHostSourcePath="@(NativeRestoredAppHostNETCore)" | ||
| AppHostDestinationDirectoryPath="$(AppHostDestinationDirectoryPath)" | ||
| AppName="$(AssemblyName).dll" |
There was a problem hiding this comment.
Probably shouldn't be hard-coding .dll here. If you really need the extension, use $(TargetExt). Why do we use the extension in the AppName?
There was a problem hiding this comment.
This is the extension of the managed assembly, i believe it will always be a ".dll"
There was a problem hiding this comment.
If it would always be a .dll, then we wouldn't have the $(TargetExt) property. We would just hard-code .dll everywhere.
Use the property.
There was a problem hiding this comment.
But again, why do we put the extension in the AppName? MyApp isn't "named" MyApp.dll.
There was a problem hiding this comment.
@eerhardt Please see my first comment on the review of this PR - I think it should not be called AppName but rather AppBinaryName.
| return; | ||
| } | ||
|
|
||
| if (AppName.Length > 1024) |
There was a problem hiding this comment.
Shouldn't this be checked against utf8-encoded byte length?
There was a problem hiding this comment.
It is not. If AppName has characters outside ASCII, it will take up more than AppName bytes.
|
|
||
| protected override void ExecuteCore() | ||
| { | ||
| var array = File.ReadAllBytes(AppHostSourcePath); |
There was a problem hiding this comment.
How big is apphost? This code seems very likely to be slow.
There was a problem hiding this comment.
This is a one time operation for every project. We have made the design decision to replace the byte stream in the host, i suppose there is no way out
On win7-x64 it is 74 kb
There was a problem hiding this comment.
Surely we can do better than copying everything in to 74 kb managed array and doing LINQ (!) over it. See @schellap's sample. Maybe it doesn't have to be quite that sophisticated, but I think the naive approach here is pushing it.
There was a problem hiding this comment.
You should move the byte[] read to be after the checks below so that it is not performed in vain.
| <EmbedAppNameInHost AppHostSourcePath="@(NativeRestoredAppHostNETCore)" | ||
| AppHostDestinationDirectoryPath="$(AppHostDestinationDirectoryPath)" | ||
| AppName="$(AssemblyName).dll" | ||
| StringToReplace="foobar" |
There was a problem hiding this comment.
Foobar is among terms that are not to be used in products. Also, the blind search and replace strategy carries risk that the mere 5 bytes occur elsewhere in the exe with completely different meaning.
| throw new BuildErrorException(Strings.FileNameIsTooLong, AppName); | ||
| } | ||
|
|
||
| var byteArray = SHA256.Create().ComputeHash(Encoding.UTF8.GetBytes(StringToReplace)); |
There was a problem hiding this comment.
Oh, I missed that we're replacing the sha of foobar, not foobar. So disregard 5 byte risk part of earlier comment. Still why are we wasting cpu resources computing this fixed value every time?
| <value>Given File Name '{0}' is longer than 1024 bytes</value> | ||
| </data> | ||
| <data name="HostIsNotNuetered" xml:space="preserve"> | ||
| <value>The App host does not have the embedded hash value of string '{0}'</value> |
There was a problem hiding this comment.
As of now, this will report 'foobar' to user which would be a world ready tenet blocker and really embarrassing if we ever have to explain this error.
What is the point of using a hash for this marker. Can't we just generate a guid?
I think the user facing message for this should not go in to the implementation details of the replacement. This is basically an invalid (corrupted?) apphost, right?
|
Btw, you wouldn't be replacing the foobar in the exe. You should replace
SHA-256 of foobar in the exe. Just use the SHA here (Btw, it is some opaque
SHA -- called out as foobar just for the unittest and comments just in case
you wanted to regenerate.)
…On Fri, Mar 10, 2017 at 5:44 PM, Rama krishnan Raghupathy < ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.NET.Sdk.targets
<#978 (comment)>:
> + <CopyToPublishDirectory>Never</CopyToPublishDirectory>
+ </AllNETCoreCopyLocalItems>
+ </ItemGroup>
+
+ <PropertyGroup Condition="'@(NativeRestoredAppHostNETCore)' != '' ">
+ <AppHostDestinationDirectoryPath>$(BaseIntermediateOutputPath)\$(TargetFramework)\$(RuntimeIdentifier)\host</AppHostDestinationDirectoryPath>
+ </PropertyGroup>
+
+ <NETSdkError Condition="'@(NativeRestoredAppHostNETCore->Count())' > 1"
+ ResourceName="MultipleFilesResolved"
+ FormatArguments="$(_DotNetAppHostExecutableName)" />
+
+ <EmbedAppNameInHost AppHostSourcePath="@(NativeRestoredAppHostNETCore)"
+ AppHostDestinationDirectoryPath="$(AppHostDestinationDirectoryPath)"
+ AppName="$(AssemblyName).dll"
+ StringToReplace="foobar"
blame @schellap <https://github.com/schellap> for this, does it make more
sense to move this string into C# or just hard code the sha value ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#978 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AKJrs8VKfJ5MqC-LlKPzdXZe3YTbmJGKks5rkfx4gaJpZM4MaAfr>
.
|
|
Also, I'd recommend a fast string replacement algorithm. I'd imagine you
want `dotnet build` to be pretty fast for large projects. See the sample in
the unit test:
https://github.com/dotnet/core-setup/blob/67f99ac9084ac132a1764251eee1157aec610bf5/test/TestUtils/AppHostExtensions.cs
You might want to do a quick perf study with a simple C# program with both
implementations and choose whatever suits best.
…On Fri, Mar 10, 2017 at 5:44 PM, Rama krishnan Raghupathy < ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.NET.Sdk.targets
<#978 (comment)>:
> + <CopyToPublishDirectory>Never</CopyToPublishDirectory>
+ </AllNETCoreCopyLocalItems>
+ </ItemGroup>
+
+ <PropertyGroup Condition="'@(NativeRestoredAppHostNETCore)' != '' ">
+ <AppHostDestinationDirectoryPath>$(BaseIntermediateOutputPath)\$(TargetFramework)\$(RuntimeIdentifier)\host</AppHostDestinationDirectoryPath>
+ </PropertyGroup>
+
+ <NETSdkError Condition="'@(NativeRestoredAppHostNETCore->Count())' > 1"
+ ResourceName="MultipleFilesResolved"
+ FormatArguments="$(_DotNetAppHostExecutableName)" />
+
+ <EmbedAppNameInHost AppHostSourcePath="@(NativeRestoredAppHostNETCore)"
+ AppHostDestinationDirectoryPath="$(AppHostDestinationDirectoryPath)"
+ AppName="$(AssemblyName).dll"
+ StringToReplace="foobar"
blame @schellap <https://github.com/schellap> for this, does it make more
sense to move this string into C# or just hard code the sha value ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#978 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AKJrs8VKfJ5MqC-LlKPzdXZe3YTbmJGKks5rkfx4gaJpZM4MaAfr>
.
|
| } | ||
|
|
||
| var placeHolder = "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"; //hash value embedded in apphost.exe | ||
| placeHolder = placeHolder.Replace("-", "").ToLower(); |
There was a problem hiding this comment.
There are no hyphens to replace in placeholder
There was a problem hiding this comment.
Nor anything to ToLower()
Why would I ever want to regenerate something opaque or why would unit test care to check that is in fact generated in any particular way? I think it will only serve to confuse folks that this is at all linked to foobar. You could have just generated 32 securely random bytes. Can we still change the marker? Policheck flags every occurrence of foobar in source code and this is just inviting folks to use it and waste cycles. |
|
As far as perf, two questions:
|
|
Regarding #1
Yes, @gkhanna79 and I did consider placing a file next to the exe in the
nupkg, but the reasoning was: it is difficult to keep track of the contract
that comes with this file and the exe no longer would be the single source
of truth and `dotnet build` can definitely place a file along side the exe
to memoize after first use.
Regarding #2
More details are here: https://github.com/dotnet/core-proposals/pull/34.
Main issue is partner signing.
…On Fri, Mar 10, 2017 at 9:46 PM, Nick Guerrera ***@***.***> wrote:
As far as perf, two questions:
1.
Why not ship a tiny data file in the package that indicates the offset
of the marker. No matter how clever, it seems wasteful to go hunting for it
on every standalone build.
2.
Why do we need to write the app name in to the apphost? What was wrong
with it just detecting the name from whatever it gets renamed to?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#978 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AKJrs-lm7bkLcnenz16qN-q-Jr-jUTHiks5rkjUcgaJpZM4MaAfr>
.
|
| throw new BuildErrorException(Strings.FileNameIsTooLong, AppName); | ||
| } | ||
|
|
||
| var placeHolder = "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"; //hash value embedded in apphost.exe |
There was a problem hiding this comment.
It will be good to add more comments around this and pointers to areas in Core-Setup to which this is connect. It will help the reader with more context.
| public string AppHostDestinationDirectoryPath { get; set; } | ||
|
|
||
| [Required] | ||
| public string AppName { get; set; } |
There was a problem hiding this comment.
It is not really AppName but AppBinaryName, right? Can we reflect it in the field naming as well?
| var placeHolder = "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"; //hash value embedded in apphost.exe | ||
| placeHolder = placeHolder.Replace("-", "").ToLower(); | ||
|
|
||
| var bytesToWrite = Encoding.UTF8.GetBytes(AppName); |
There was a problem hiding this comment.
I would also suggest writing a comment describing what is happening in the next set of statements for the reader to get context.
|
|
||
| <!-- publishing self-contained apps should publish the native host as $(AssemblyName).exe --> | ||
| <RenamePublishedHost Condition=" '$(RenamePublishedHost)' == '' and '$(OutputType)' == 'Exe' and '$(RuntimeIdentifier)' != ''">true</RenamePublishedHost> | ||
| <DeployAppHost Condition=" '$(DeployAppHost)' == '' and '$(OutputType)' == 'Exe' and '$(RuntimeIdentifier)' != ''">true</DeployAppHost> |
There was a problem hiding this comment.
Correct, here and other places in the code '$(RuntimeIdentifier)' != '' means it is a standalone app.
| DeployAppHost | ||
|
|
||
| Renames the dotnet native host to match the application's name. | ||
| Deploys the host to run the app and ensures it matches the app name |
There was a problem hiding this comment.
We should update this to reflect that this applies to standalone apps only.
|
|
||
| referencerAsset.Restore(testProject.Name); | ||
| var publishCommand = new PublishCommand(Stage0MSBuild, Path.Combine(referencerAsset.TestRoot, testProject.Name)); | ||
| var publishResult = publishCommand.Execute($"/p:RuntimeIdentifier={rid}"); |
There was a problem hiding this comment.
You don't need to pass /p:RuntimeIdentifier={rid} to publish. That value is already hard-coded into the .csproj up above. You can just say .Execute().
| "Hello.pdb", | ||
| "Hello.deps.json", | ||
| "Hello.runtimeconfig.json", | ||
| $"{libPrefix}coreclr{Constants.DynamicLibSuffix}", |
There was a problem hiding this comment.
Why do we use Constants.DynamicLibSuffix, but we hard-code our own prefix?
There was a problem hiding this comment.
There is no prefix in that class
| .Pass() | ||
| .And | ||
| .HaveStdOutContaining("Hello from a netcoreapp2.0.!"); | ||
| } |
There was a problem hiding this comment.
Do you also want to ensure that the executable has the app name embedded in it correctly?
There was a problem hiding this comment.
if it is not the apphost will fail. I was wondering should i add a knob to turn off the EmbedAppNameInHost and add a test case to turn of the knob and see the test fail, i decided against it as this is testing the host and we already do that in core-setup
|
@dotnet-bot test this please |
|
@eerhardt @gkhanna79 @eerhardt i have addressed the comments |
| public string AppBinaryName { get; set; } | ||
|
|
||
| [Output] | ||
| public string ModifiedAppHostPath |
There was a problem hiding this comment.
Nit: Can be public string ModifiedAppHostPath { get; private set; }
| get { return _modifiedAppHostPath; } | ||
| } | ||
|
|
||
| private static string placeHolder = "c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2"; //hash value embedded in default apphost executable |
There was a problem hiding this comment.
Nit: placeholder is one valid word so use placheholder casing (no capital H)
|
|
||
| protected override void ExecuteCore() | ||
| { | ||
|
|
|
|
||
| SearchAndReplace(array, bytesToSearch, bytesToWrite); | ||
|
|
||
|
|
| SearchAndReplace(array, bytesToSearch, bytesToWrite); | ||
|
|
||
|
|
||
| if ( !Directory.Exists(destinationDirectory)) |
There was a problem hiding this comment.
nit: extra whitespace next to paren
|
|
||
| } | ||
| } | ||
|
|
| } | ||
|
|
||
| } | ||
|
|
| <value>The App host has been modified, it is missing the default embedded value '{0}'</value> | ||
| </data> | ||
| <data name="FileNameIsTooLong" xml:space="preserve"> | ||
| <value>Given File Name '{0}' is longer than 1024 bytes</value> |
There was a problem hiding this comment.
file name (no caps, it's not a proper noun).
| } | ||
|
|
||
| var array = File.ReadAllBytes(AppHostSourcePath); | ||
| var bytesToSearch = Encoding.UTF8.GetBytes(placeHolder); |
There was a problem hiding this comment.
We shouldn't need to re-encode the placeholder every time, why not just have a static readonly byte[] for that.
| throw new BuildErrorException(Strings.FileNameIsTooLong, AppBinaryName); | ||
| } | ||
|
|
||
| var array = File.ReadAllBytes(AppHostSourcePath); |
There was a problem hiding this comment.
I'm still concerned about reading the entire file in to one big managed array every time.
There was a problem hiding this comment.
This is a one time per project's lifetime. We can revisit it later, will file a bug for it
| } | ||
|
|
||
| // See: https://en.wikipedia.org/wiki/Knuth%E2%80%93Morris%E2%80%93Pratt_algorithm | ||
| private static int[] ComputeKMP_FailureFunction(byte[] pattern) |
There was a problem hiding this comment.
Don't name non-test methods with underscores.
This doesn't need to be re-computed every time either.
| <value>Specify a RuntimeIdentifier</value> | ||
| </data> | ||
| <data name="AppHostHasBeenModified" xml:space="preserve"> | ||
| <value>The App host has been modified, it is missing the default embedded value '{0}'</value> |
There was a problem hiding this comment.
This will be very rare so I think the message should help identify the problem better. Saying "modified" is a bit misleading as it could also be a bug where we get out of sync with apphost changes.
I suggest something like "Unable to use '{0}' as application host executable as it does not contain the expected placeholder byte sequence '{1}' that would mark where the application name would be written."
{0} would be the full path to the file so we can see package version, etc. if someone actually hits this and asks for support.
|
I'm OK postponing the perf changes, but I'd like an issue to be filed to look at that. Please clean up the whitespace and the error message before merging. |
* Update dependencies from https://github.com/dotnet/arcade build 20191001.4 - Microsoft.DotNet.Arcade.Sdk - 5.0.0-beta.19501.4 * Update dependencies from https://github.com/dotnet/arcade build 20191002.11 - Microsoft.DotNet.Arcade.Sdk - 5.0.0-beta.19502.11
@gkhanna79 @eerhardt @nguerrera PTAL