diff --git a/docs/design/tools/illink/task-cache.md b/docs/design/tools/illink/task-cache.md index 54c0a8c59e2fa9..c6f8dd5e67c87f 100644 --- a/docs/design/tools/illink/task-cache.md +++ b/docs/design/tools/illink/task-cache.md @@ -1,94 +1,112 @@ -# ILLink task cache: v1 decisions - -These are design decisions. Eligible task invocations compute a content-based key, restore a cached output directory on a hit, or run ILLink and store successful results. - -The feature is experimental. Its configuration and behavior may change or be removed without notice; it is not a supported compatibility surface. - -## Scope and configuration - -- Cache complete ILLink invocations in the MSBuild task, not in the linker command-line implementation. -- Keep v1 simple. Reuse results at the same input/output paths; defer cross-worktree reuse and retain absolute paths in cache identity. -- Enable caching only through the `ILLINK_EXPERIMENTAL_CACHE` environment variable set to `true` (case-insensitive). Unset, empty, or `false` disables caching; invalid boolean values disable caching with a diagnostic. Disabled caching performs no cache I/O, even if a cache directory already exists. -- Use the optional `ILLINK_EXPERIMENTAL_CACHE_PATH` environment variable for the directory override. Unset or empty selects the platform default. Selecting a directory does not enable caching, and path values are never interpreted as booleans. -- Use platform-specific default directories: `$XDG_CACHE_HOME/illink` on Linux/other XDG Unix, falling back to `$HOME/.cache/illink`; `$HOME/Library/Caches/illink` on macOS; `LocalApplicationData/illink` on Windows. -- Resolve defaults inside `ILLinkCache.TryCreate` through a private helper, not in targets. Resolve relative overrides against the current working directory. -- Ignore relative `XDG_CACHE_HOME` values. If no usable user location is available, skip caching with a diagnostic rather than fall back to the working directory or shared temporary storage. -- Read configuration from the task process's environment on each invocation. Set it before starting MSBuild; do not change process-wide environment settings during parallel builds. Reused build processes must receive the intended environment. There are no cache task parameters, MSBuild property wiring, or linker command-line switches. -- Do not promise stability or migration support for the on-disk format. - -## API shape - -```csharp -internal sealed class ILLinkCache -{ - internal static ILLinkCache? TryCreate(string? cacheDirectory, TaskLoggingHelper log); - internal bool TryRestore(string inputHash, string outputDirectory, DateTime outputTimestampUtc); - internal void Store(string inputHash, string outputDirectory); -} -``` - -- The task owns input identity and execution. Its private `TryComputeCacheKey` combines command-line and response-file arguments with input-content and toolchain identity; inability to compute a key bypasses caching with a diagnostic. -- `TryCreate` resolves the cache directory and returns null with a diagnostic if initialization cannot proceed. -- `TryRestore` combines lookup and restoration. True means complete restoration. Missing entries and expected I/O, access, or invalid-cache-data failures return false with appropriate logging and fall back to normal linking. Copies overwrite existing files and are not rolled back if a later copy fails. Unexpected exceptions propagate. -- `Store` is best-effort. Lock contention and existing entries are normal skips; expected cache I/O failures are logged without failing a successful link. Unexpected exceptions propagate. -- Expected restore/store failures are `IOException`, `UnauthorizedAccessException`, `InvalidDataException`, and `FormatException`, not every exception. Invalid helper arguments remain programming errors. -- After normal task validation, resolve the cache directory before computing the key so an unusable configuration does not trigger input inspection or hashing. If resolution succeeds, compute the key once and attempt restoration. On a hit, skip linking; otherwise link normally and store only with exit code zero and no logged errors. A failed restore never substitutes for a successful fallback link. - -## Execution and publication - -- Cache operations do not support cancellation in v1, including key computation, validation, restoration, and storage. Assume they are fast relative to actual linker execution, and keep synchronous filesystem operations without cancellation polling or chunked cancellable I/O. A cancellation request may arrive while these operations finish, and a cache hit may return success without observing it. Normal linker execution retains `ToolTask`'s existing subprocess cancellation behavior. -- Check the cache before normal tool execution. On a hit, validate the complete entry, then delete/recreate the destination and copy the cached tree. On any miss or restore failure, the normal execution path deletes/recreates the directory before running ILLink; no separate rollback or failure-cleanup path is needed. -- The task owns its output directory, including when caching is disabled. Normal-execution cleanup failures fail the task. Rely on the supplied `OutputDirectory` parameter without additional task path/input validation; callers must provide a dedicated directory. The command-line linker retains its existing nonempty-directory behavior. -- ILLink writes to its normal output directory. After successful linking, copy results into the cache; do not redirect linker output to a temporary directory. -- Let cache misses compute independently. Scheduling and execution times are nondeterministic, and steps can fail or be aborted; one computation must not block another's progress. -- Use a unique cache staging directory for each store attempt, including attempts with the same input hash. -- Acquire a per-input-hash writer lock, scoped to the cache directory, after linking and before staging copies. -- Acquire the lock nonblocking. If busy, skip storing rather than wait; this avoids duplicate cache-copy work, not duplicate linking. The losing task keeps its own computed outputs and does not wait or retry if the winning writer later fails or is aborted. -- Under the lock, recheck whether the entry already exists. If absent, populate staging and publish with an atomic, same-filesystem directory rename without overwriting an existing entry. -- Rely on the documented `Directory.Move` contract that an existing destination causes `IOException`, even if that destination is empty. The writer lock avoids duplicate staging work; it is not a workaround for publication semantics. Do not add platform-specific interop or require non-empty entries to compensate for runtime/filesystem bugs. Atomic visibility does not imply power-loss durability. -- Create cache directories lazily in `Store`, after acquiring the writer lock and confirming the entry is absent. `TryCreate` only resolves the path; `TryRestore` treats a missing directory as a miss. Directory-creation failures are logged without failing a successful link. -- Keep published outputs and manifests immutable and read them without locks. Write a UTC `last-used` timestamp in staging before publication and refresh it after a successful restore. Marker refreshes are best-effort overwrites; usage-update failures are logged without failing linking. An interrupted write may leave an invalid marker, which purge handles using the entry directory's creation time. -- Use a named `System.Threading.Mutex` derived from the resolved cache-directory path and input hash. An abandoned mutex grants ownership; recheck the entry and use fresh staging. This coordinates processes on one machine, not writers on different machines sharing a network directory. -- Support concurrent task invocations with distinct output directories sharing a cache directory. Concurrent linking to the same output directory is unsupported. -- Treat missing or unreadable cache data as a failed restore, never a successful hit. -- Start without replaying warnings or other linker diagnostics on cache hits. This is an intentional v1 limitation. - -## Input identity and eligibility - -- Use SHA-256 over a versioned, length-delimited description. Preserve argument ordering and absolute paths, including the working and output directories. -- Hash the contents of all supplied assemblies, reference assemblies, and root descriptors. Include adjacent PDBs, MDBs, configuration files, and satellite assemblies, even when symbol emission is disabled. Follow assembly metadata file entries for linked resources and additional modules. Optional-file additions and removals change the key; timestamps alone do not. -- Hash the entire linker assembly on every invocation, without loading it or using its module version ID (MVID). Content changes invalidate the cache even when the MVID, file length, and timestamp are unchanged. Do not scan its directory. Bundled dependencies and configuration are not fingerprinted: changes to Cecil or deployment configuration require a cleared/isolated cache or disabled caching unless the linker binary also changes. -- Continue hashing the task assembly and selected dotnet host executable. Do not enumerate or hash the host's installed hostfxr/shared-runtime trees: doing so makes lookup cost scale with every installed framework/version, including ones unrelated to the link. Runtime/tool files supplied as linker inputs remain content-hashed. -- Defer identifying the runtime that executes ILLink; do not include a runtime version or framework description in the key. The task runs in MSBuild, so its CoreLib informational version and framework description are not reliable identities for the child linker's runtime. The runtimeconfig describes requested frameworks and roll-forward policy, not the exact resolved CoreLib. Finding that CoreLib would require resolved-runtime information from the caller, host-resolution plumbing, or an additional process; defer that complexity and startup cost. -- This deliberately accepts that runtime installation changes not represented by other keyed inputs may reuse an entry; callers needing invalidation for those changes must clear or isolate the cache. -- Include platform, architecture, and culture. Do not enumerate or hash inherited environment variables, or bypass caching based on them. This deliberately keeps v1 simple rather than maintaining a partial list of runtime/loader settings. Environment changes are reflected only when they change other keyed inputs, such as the selected host path or culture. Callers must disable caching when inherited settings introduce otherwise untracked dependencies, affect outputs, or require tool-execution side effects (for example, startup hooks or profilers). -- Bypass caching for arbitrary `ExtraArgs`, custom steps/data, dependency-dump options, and explicit `ToolTask.EnvironmentVariables` overrides. These are caller-supplied inputs that v1 does not model in its key and can introduce undeclared dependencies, external outputs, or side effects. The supported `ExtraArgs` shape for runtime's library builds is `--ignore-link-attributes true` followed by zero or more `--link-attributes FILE`, `--substitutions FILE`, or `-d DIRECTORY` pairs. Hash every referenced XML file and every top-level `.dll`, `.exe`, and `.winmd` candidate in each search directory, including assembly sidecars/modules/resources. Preserve directory order in the arguments and leave linker resolution unchanged; adding, changing, or removing candidates invalidates the key, including shadowed candidates. An unreadable directory or uninspectable candidate bypasses caching. All other SDK options passed through `ExtraArgs` remain subject to the bypass. -- Log and bypass caching if required files cannot be read or an input assembly cannot be inspected, including PE images without managed metadata. Normal linking determines whether those inputs are acceptable. Input files must remain stable during hashing and linking. - -## Entry layout - -- Store entries under `/v1//`, with a binary `manifest`, an `outputs/` tree, and a `last-used` UTC timestamp in invariant round-trip (`O`) text format. -- The manifest records the format version, output paths, lengths, and SHA-256 content hashes. Validate it and every listed file before starting restoration. -- Copy the complete output tree without special-casing filenames. The MSBuild target owns `.semaphore` (normally `linked.semaphore`) and invalidates it before the incremental skip decision when that directory is missing. There is no special migration or exclusion for old in-directory `Link.semaphore` files; ordinary whole-directory cleanup and caching apply to them. Empty directories are not recorded. -- The restore helper replaces the destination tree after entry validation. Copy failures can leave partial results, which the task's normal execution cleanup removes before fallback linking. Publication is atomic; restoration is not. -- Assume ILLink outputs are not symbolic links; do not check for them. Cache and output directories must not contain each other. -- Stamp restored files with the UTC timestamp captured before input-key inspection. This avoids stale cache timestamps in downstream incremental checks without making outputs appear newer than inputs changed during inspection or restoration. - -## Cache lifetime - -- No automatic eviction or size policy during linking. Users may manually wipe the cache directory or explicitly run the experimental `dotnet-illink-cache purge` tool. -- Do not repair or replace corrupt published entries. Restoration falls back to linking, but storage still skips an existing entry until the user removes it. -- No `created` bookkeeping marker. The `last-used` marker records usage for explicit post-build maintenance. -- Purge requires an explicit cache root and UTC cutoff. Only published SHA-256 entry directories in `v1` are considered; delete entries whose last-used time is strictly older than the cutoff and retain equality. Leave staging directories and other layouts alone. Report per-entry maintenance errors and continue, then return nonzero if maintenance failed. -- For a missing, malformed, non-UTC, or unreadable marker, report the fallback and use the entry directory's creation time instead. This may purge a recently used entry or retain a stale one if restoring a cache snapshot resets directory creation times. Do not migrate or repair the marker. -- Purge must not run during any build using the cache. Do not add purge locks or change reader locking; coordination with active readers/writers remains a separate follow-up, as does protection against manual deletion. -- Only the `CoreLib_Libraries_AnalyzersLinker` CI job opts into ILLink caching through `setup-build-caches.yml`, on Windows and Unix x64/arm64. Roslyn and sccache remain gated by platform. Restore a job-private Azure cache, record build start, run linking and all post-build steps, then purge against that timestamp before uploading the next snapshot. This job explicitly enables `RunILLinkInBuild` and builds the `tools.illink` subset for the maintenance tool; building `ILLink.Tasks` through a library project reference does not build it. Purge only after a successful build; skip maintenance when no versioned cache directory exists and report a missing tool. Preserve tool diagnostics but do not fail the build for maintenance errors. Only purge once all users have exited; incremental skips and unreached work in failed builds do not refresh usage. See the [CI recipe and wiring](../../../tools/illink/illink-tasks.md#ci-purge-ordering). -- Attempt best-effort cleanup of the current attempt's unpublished staging directory. Log cleanup I/O/access failures without failing a successful link; do not scan or delete other attempts' staging directories. Correctness must tolerate leftovers after crashes. - -## Open follow-ups - -- Revisit identifying the resolved runtime/CoreLib that executes ILLink without adding substantial cache-lookup overhead. -- Revisit symbolic-link handling when generalizing beyond ILLink outputs. -- Revisit cross-process shared-reader/exclusive-purge coordination if concurrent maintenance becomes necessary. -- Reconsider diagnostic replay, including missing console/binlog warnings and effects on external warning-as-error policies. -- Examine manual wiping racing reads, including safe fallback after partially restoring output files. +# ILLink task cache integration + +ILLink opts eligible task invocations into the +[shared invocation cache](../../../../src/tasks/Common/InvocationCache/README.md). This document +covers ILLink's input discovery, output ownership, and deliberate omissions; the shared README owns +hashing, storage, restoration, and concurrency. The feature is experimental and may change or be +removed without notice. + +## Configuration and task boundary + +Cache complete invocations in the MSBuild task, not in the linker CLI. Enable with +`ILLINK_EXPERIMENTAL_CACHE=true` and optionally override storage with +`ILLINK_EXPERIMENTAL_CACHE_PATH`. These are process environment variables, not task parameters or +MSBuild properties. Set them before starting MSBuild and do not change them during parallel builds; +reused processes must receive the intended environment. + +The task supplies display name `ILLink` to the shared configuration. The default is `illink` under the +platform cache location; a path alone never enables caching. See +[task usage](../../../tools/illink/illink-tasks.md#experimental-task-cache) for configuration and +maintenance commands. + +`ExecuteTool` calls `InvocationCache.ExecuteCached` with parameterless callbacks for the task's +`DescribeInvocation` and `ExecuteNormally` methods. The description supplies input paths, effective +arguments/context, and one `InvocationOutputs.FromDirectory` declaration. Eligibility and metadata +inspection stay in ILLink; content hashing and cache operations stay in the shared layer. + +## Input discovery and eligibility + +Hash all supplied assemblies, reference assemblies, and root descriptors. Include adjacent PDBs, +MDBs, configuration files, and satellite assemblies even when symbols are disabled. Follow assembly +metadata file entries for linked resources and additional modules. Adding or removing a sidecar +changes identity. + +The inventory includes the task assembly, selected dotnet host executable, and entire linker +assembly. Hash the linker without loading it or relying on MVID, timestamps, or length; do not scan +its deployment directory or the host's installed runtime trees. + +Bundled linker dependencies such as Cecil and deployment configuration are not fingerprinted. +Changes to those files require a cleared/isolated cache or disabled caching unless the linker binary +also changes. + +The resolved runtime/CoreLib executing the child linker is deliberately not identified. MSBuild's +runtime identity does not establish the child's, and runtimeconfig roll-forward settings do not +identify the resolved CoreLib. Resolving it would require additional host information or another +process. Runtime installation changes alone therefore need not invalidate entries; callers +requiring that distinction must clear or isolate storage. + +Inherited environment variables are not enumerated, hashed, or used as a bypass list. Platform, +architecture, and culture are part of the shared key, but callers must disable caching when inherited +settings introduce untracked dependencies, affect outputs, or require side effects such as startup +hooks or profiling. + +Bypass arbitrary `ExtraArgs`, custom steps/data, dependency-dump options, and explicit +`ToolTask.EnvironmentVariables` overrides. These can introduce undeclared inputs, outputs, or side +effects. + +For runtime library builds, support `--ignore-link-attributes true` followed by zero or more +`--link-attributes FILE`, `--substitutions FILE`, or `-d DIRECTORY` pairs. Require the XML files to be +readable. Inventory every top-level `.dll`, `.exe`, and `.winmd` candidate in each search directory, +including sidecars, modules, and resources. Preserve directory order and leave linker resolution +unchanged; adding, changing, or removing even a shadowed candidate invalidates identity. Other SDK +options supplied through `ExtraArgs` remain unsupported. + +Unreadable directories, uninspectable assemblies (including PE images without managed metadata), and +other input-read failures bypass caching with diagnostics. Missing declared files have explicit +absence markers; normal linking decides whether an invocation can succeed. Inputs must remain stable +throughout discovery, hashing, and linking. + +## Output preparation and diagnostics + +The task owns a dedicated `OutputDirectory`, including when caching is disabled. `ExecuteNormally` +deletes and recreates it exactly once using ordinary filesystem operations, then executes ILLink. +Cleanup errors fail the task. Do not add cache-aware preparation callbacks or make normal execution +depend on cache eligibility. The linker CLI retains its existing nonempty-directory behavior. + +On a hit, the shared layer validates the entry and replaces the owned directory. On a miss or failed +restore, normal execution prepares it before linking. ILLink writes directly to its ordinary +destination; successful results are copied into storage afterward, not compiled into cache staging. + +For eligible cache-enabled invocations, the cache and output directories must neither equal nor +contain each other. Invalid overlap fails before cleanup rather than falling back to uncached +linking. + +Copy the complete output tree without filename exceptions. The MSBuild target owns +`.semaphore` (normally `linked.semaphore`) and invalidates it before its incremental +skip decision when the directory is missing. Old in-directory `Link.semaphore` files receive no +special migration or exclusion. Assume outputs are not symbolic links. + +Hits do not replay linker warnings or other diagnostics. This can omit console/binlog warnings and +affect external warning-as-error policies. Callers needing those diagnostics or execution side effects +must disable caching. Normal tool execution retains `ToolTask` subprocess cancellation behavior; +shared cache operations do not observe cancellation. + +## Maintenance and CI integration + +Use the shared [maintenance CLI](../../../../src/tools/BuildCache/README.md) with `--cache illink`, +or select an explicit directory. Command parsing, cache-name mappings, packaging, and installation +are documented there; purge semantics are in the shared cache README. + +The `CoreLib_Libraries_AnalyzersLinker` CI job opts in through `setup-build-caches.yml` on Windows and +Unix x64/arm64. Restore a job-private Azure cache, record build start, run the build and all users, +then purge against that timestamp before uploading the next snapshot. This job enables +`RunILLinkInBuild` and builds `tools.illink` to obtain the maintenance tool; a library project +reference to `ILLink.Tasks` does not build it. + +Purge only after successful builds and after all users exit. Skip absent versioned storage and report +a missing maintenance tool. Preserve tool diagnostics without failing the build for maintenance +errors. Incremental skips and unreached work do not refresh usage. See the +[CI recipe and wiring](../../../tools/illink/illink-tasks.md#ci-purge-ordering). + +ILLink-specific follow-ups are resolved child-runtime identity and diagnostic replay. Shared +maintenance coordination and restoration limitations belong in the shared README. diff --git a/docs/tools/illink/README.md b/docs/tools/illink/README.md index 864c6fed5e0f68..db975c2f6e6a53 100644 --- a/docs/tools/illink/README.md +++ b/docs/tools/illink/README.md @@ -15,3 +15,4 @@ View the official trimming docs at [learn.microsoft.com](https://learn.microsoft ## Design Docs - [Reflection Handling](../../design/tools/illink/reflection-flow.md) - [Task Cache](../../design/tools/illink/task-cache.md) +- [Shared Invocation Cache](../../../src/tasks/Common/InvocationCache/README.md) diff --git a/docs/tools/illink/illink-tasks.md b/docs/tools/illink/illink-tasks.md index 7e11f70f63b0f0..ca525e084aeab7 100644 --- a/docs/tools/illink/illink-tasks.md +++ b/docs/tools/illink/illink-tasks.md @@ -53,6 +53,9 @@ The cache helper resolves an unset or empty directory override to `$XDG_CACHE_HO `$HOME/.cache/illink`) on Linux/other XDG Unix, `$HOME/Library/Caches/illink` on macOS, and `LocalApplicationData/illink` on Windows. Relative `XDG_CACHE_HOME` values are ignored; explicit relative cache directories are resolved against the working directory. +Keep the cache and output directories separate: neither may equal or contain the other. +For eligible cache-enabled invocations, overlap fails the task before output cleanup; +it does not fall back to uncached linking. Eligible invocations use a key covering arguments, input-file and sidecar contents, the task assembly, the selected dotnet host executable, and the entire ILLink assembly. @@ -80,15 +83,17 @@ fall back to normal linking. See the ### Cache maintenance -The experimental `dotnet-illink-cache` .NET tool can purge entries that were not used recently: +The shared experimental `dotnet-build-cache` .NET tool can purge entries that were not used recently: ```sh -dotnet illink-cache purge --cache-directory /path/to/cache --before 2026-10-01T12:00:00Z +dotnet build-cache purge --cache illink --before 2026-10-01T12:00:00Z ``` -The directory and UTC cutoff are required; maintenance does not use the task's opt-in or -directory environment variables. Use the tool built from the same revision as the task. -See the [tool README](../../../src/tools/illink/src/ILLink.CacheTool/README.md) for local +The UTC cutoff is required. `--cache illink` uses `ILLINK_EXPERIMENTAL_CACHE_PATH` or the +task's platform default. Alternatively, use `--cache-directory /path/to/cache` on its own +to select a directory directly. Maintenance ignores the task's boolean opt-in and +never creates missing directories. Use the tool built from the same revision as the task. +See the [tool README](../../../src/tools/BuildCache/README.md) for local packaging and installation. Each entry has a `last-used` timestamp, initialized during publication and refreshed after @@ -118,7 +123,7 @@ or created by that build: 3. Run the build successfully and wait for all processes using the cache to finish. 4. Run the maintenance tool from the same checkout, with maintenance failures reported but nonfatal: ```sh - if ! dotnet illink-cache purge \ + if ! dotnet build-cache purge \ --cache-directory "$ILLINK_EXPERIMENTAL_CACHE_PATH" \ --before "$ILLINK_CACHE_BUILD_START"; then echo "Warning: ILLink cache purge failed; see the tool diagnostics above." >&2 diff --git a/eng/Subsets.props b/eng/Subsets.props index 539be327d11cb2..1349b77fbfbd0f 100644 --- a/eng/Subsets.props +++ b/eng/Subsets.props @@ -148,7 +148,7 @@ $(DefaultLibrariesSubsets)+libs.tests - tools.illink+tools.cdac + tools.buildcache+tools.illink+tools.cdac host.native+host.tools+host.pkg $(DefaultHostSubsets)+host.pretest+host.tests @@ -173,7 +173,7 @@ $(AllSubsetsExpansion)+clr.paltests+clr.paltestlist+clr.hosts+clr.jit+clr.alljits+clr.alljitscommunity+clr.spmi+clr.corelib+clr.nativecorelib+clr.tools+clr.toolstests+clr.r2rtests+clr.packages $(AllSubsetsExpansion)+linuxdac+alpinedac $(AllSubsetsExpansion)+mono.runtime+provision.emsdk+mono.aotcross+mono.corelib+mono.manifests+mono.packages+mono.tools+mono.wasmruntime+mono.wasiruntime+mono.wasmworkload+mono.mscordbi - $(AllSubsetsExpansion)+tools.illink+tools.cdac+tools.illinktests+tools.cdactests+tools.cdacdumptests + $(AllSubsetsExpansion)+tools.buildcache+tools.illink+tools.cdac+tools.buildcachetests+tools.illinktests+tools.cdactests+tools.cdacdumptests $(AllSubsetsExpansion)+host.native+host.pkg+host.tools+host.pretest+host.tests $(AllSubsetsExpansion)+libs.native+libs.sfx+libs.oob+libs.pretest+libs.tests $(AllSubsetsExpansion)+packs.product+packs.installers+packs.tests @@ -258,6 +258,8 @@ + + @@ -567,8 +569,15 @@ + + + + + + + + - @@ -577,8 +586,6 @@ - Path.Combine(Root, VersionDirectory); + internal StringWriter Output { get; } = new(); + internal StringWriter Error { get; } = new(); + + internal TestCache(string versionDirectory = InvocationCacheEntry.VersionDirectory) + { + VersionDirectory = versionDirectory; + Directory.CreateDirectory(Entries); + } + + internal string AddEntry(char key, DateTimeOffset lastUsed) + { + string entry = Path.Combine(Entries, new string(key, 64)); + Directory.CreateDirectory(Path.Combine(entry, "outputs")); + File.WriteAllText(Path.Combine(entry, "outputs", "app.dll"), "cached output"); + File.WriteAllText(Path.Combine(entry, InvocationCacheEntry.LastUsedFileName), lastUsed.ToString("O", CultureInfo.InvariantCulture)); + return entry; + } + + internal int Run(DateTimeOffset? cutoff = null) => + InvocationCachePurge.Run(Root, VersionDirectory, "Test", cutoff ?? Cutoff, Output, Error); + + public void Dispose() + { + Output.Dispose(); + Error.Dispose(); + Directory.Delete(Root, recursive: true); + } + } +} diff --git a/src/tasks/Common/InvocationCache.Tests/InvocationCacheTests.cs b/src/tasks/Common/InvocationCache.Tests/InvocationCacheTests.cs new file mode 100644 index 00000000000000..935a6076b8a05a --- /dev/null +++ b/src/tasks/Common/InvocationCache.Tests/InvocationCacheTests.cs @@ -0,0 +1,1215 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.Collections; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Reflection; +using System.Runtime.ExceptionServices; +using System.Security.Cryptography; +using System.Threading; +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; +using Microsoft.DotNet.RemoteExecutor; +using Xunit; + +namespace Microsoft.NET.Build.Tasks.Tests; + +public class InvocationCacheTests +{ + private const string EnabledEnvironmentVariable = "INVOCATION_CACHE_TEST_ENABLED"; + private const string PathEnvironmentVariable = "INVOCATION_CACHE_TEST_PATH"; + private static readonly InvocationCacheConfiguration s_cacheConfiguration = + new("Test", EnabledEnvironmentVariable, PathEnvironmentVariable); + + [Fact] + public void CacheConfigurationDoesNotCaptureEnvironmentValues() => RemoteExecutor.Invoke(static () => + { + using var test = new CacheTestDirectory(); + var configuration = new InvocationCacheConfiguration("Test", EnabledEnvironmentVariable, PathEnvironmentVariable); + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "false"); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, Path.Combine(test.Root, "first")); + Assert.Null(InvocationCache.TryCreateFromEnvironment(test.Log, configuration)); + + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "true"); + Assert.Equal(Path.Combine(test.Root, "first"), + InvocationCache.TryCreateFromEnvironment(test.Log, configuration)!.CacheDirectory); + + Environment.SetEnvironmentVariable(PathEnvironmentVariable, Path.Combine(test.Root, "second")); + Assert.Equal(Path.Combine(test.Root, "second"), + InvocationCache.TryCreateFromEnvironment(test.Log, configuration)!.CacheDirectory); + Assert.False(Directory.Exists(Path.Combine(test.Root, "first"))); + Assert.False(Directory.Exists(Path.Combine(test.Root, "second"))); + }).Dispose(); + + [Theory] + [InlineData(null, false, false)] + [InlineData("", false, false)] + [InlineData("false", false, false)] + [InlineData("FALSE", false, false)] + [InlineData("true", true, false)] + [InlineData("TrUe", true, false)] + [InlineData("1", false, true)] + [InlineData("invalid", false, true)] + public void CacheEnvironmentConfiguration(string? setting, bool enabled, bool invalid) + { + RemoteExecutor.Invoke(static (value, expectedEnabled, expectedInvalid) => + { + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, value == "" ? null : value); + string directory = Path.Combine(Path.GetTempPath(), "invocation-cache-" + Guid.NewGuid().ToString("N")); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, directory); + var buildEngine = new CacheBuildEngine(); + var task = new CacheTask { BuildEngine = buildEngine }; + InvocationCache? cache = InvocationCache.TryCreateFromEnvironment(task.Log, s_cacheConfiguration); + + Assert.Equal(bool.Parse(expectedEnabled), cache is not null); + if (cache is not null) + Assert.Equal(directory, cache.CacheDirectory); + if (bool.Parse(expectedInvalid)) + { + BuildMessageEventArgs message = Assert.Single(buildEngine.Messages); + Assert.Equal($"Test caching is disabled because {EnabledEnvironmentVariable} must be 'true' or 'false'.", message.Message); + Assert.Equal(MessageImportance.Low, message.Importance); + } + else + { + Assert.Empty(buildEngine.Messages); + } + Assert.False(Directory.Exists(directory)); + }, setting ?? "", enabled.ToString(), invalid.ToString()).Dispose(); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("true")] + [InlineData("false")] + [InlineData("cache directory")] + public void CacheEnvironmentDirectory(string? directory) + { + RemoteExecutor.Invoke(static value => + { + string? path = value == "" ? null : value; + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "true"); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, path); + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + + InvocationCache? cache = InvocationCache.TryCreateFromEnvironment(task.Log, s_cacheConfiguration); + + Assert.NotNull(cache); + Assert.Equal(InvocationCache.TryCreate(path, task.Log, "Test")!.CacheDirectory, cache.CacheDirectory); + }, directory ?? "").Dispose(); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + public void CacheUsesPlatformDefaultDirectory(string? cacheDirectory) + { + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + string expectedDirectory; + if (OperatingSystem.IsWindows()) + { + expectedDirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "test"); + } + else if (OperatingSystem.IsMacOS()) + { + expectedDirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), "Library", "Caches", "test"); + } + else + { + string? cacheHome = Environment.GetEnvironmentVariable("XDG_CACHE_HOME"); + expectedDirectory = !string.IsNullOrEmpty(cacheHome) && Path.IsPathRooted(cacheHome) + ? Path.Combine(cacheHome, "test") + : Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".cache", "test"); + } + + InvocationCache? cache = InvocationCache.TryCreate(cacheDirectory, task.Log, "Test"); + + Assert.NotNull(cache); + Assert.Equal(Path.GetFullPath(expectedDirectory), cache.CacheDirectory); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void CacheDirectoryOverrideDoesNotCreateDirectory(bool absolutePath) + { + string directoryName = "invocation-cache-" + Guid.NewGuid().ToString("N"); + string cacheDirectory = absolutePath ? Path.Combine(Path.GetTempPath(), directoryName) : directoryName; + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + + InvocationCache? cache = InvocationCache.TryCreate(cacheDirectory, task.Log, "Test"); + + Assert.NotNull(cache); + Assert.Equal(Path.GetFullPath(cacheDirectory), cache.CacheDirectory); + Assert.False(Directory.Exists(cache.CacheDirectory)); + } + + [Fact] + public void InvalidCacheDirectoryBypassesCaching() + { + var buildEngine = new CacheBuildEngine(); + var task = new CacheTask { BuildEngine = buildEngine }; + + Assert.Null(InvocationCache.TryCreate("invalid\0path", task.Log, "Test")); + BuildMessageEventArgs message = Assert.Single(buildEngine.Messages); + Assert.StartsWith("Test caching is disabled because the cache directory 'invalid\0path' could not be resolved:", message.Message); + Assert.Equal(MessageImportance.Low, message.Importance); + } + + [Fact] + public void CacheDirectoryFileBypassesCaching() + { + using var test = new CacheTestDirectory(); + File.WriteAllText(test.Cache.CacheDirectory, "not a directory"); + + Assert.Null(InvocationCache.TryCreate(test.Cache.CacheDirectory, test.Log, "Test")); + + BuildMessageEventArgs message = Assert.Single(test.BuildEngine.Messages); + Assert.Contains($"Cache location is a file: '{test.Cache.CacheDirectory}'.", message.Message); + Assert.Equal("not a directory", File.ReadAllText(test.Cache.CacheDirectory)); + } + + [Theory] + [InlineData("ILLink", "illink")] + [InlineData("Crossgen2", "crossgen2")] + public void CacheDirectoryNameUsesInvariantToolName(string toolName, string directoryName) + => RemoteExecutor.Invoke(static (name, expectedDirectory) => + { + CultureInfo.CurrentCulture = new CultureInfo("tr-TR"); + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + + InvocationCache? cache = InvocationCache.TryCreate(null, task.Log, name); + + Assert.NotNull(cache); + Assert.Equal(expectedDirectory, Path.GetFileName(cache.CacheDirectory)); + }, toolName, directoryName).Dispose(); + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("relative")] + [InlineData("absolute")] + public void CacheDefaultDirectoryHonorsPlatformAndXdgHome(string? setting) + { + RemoteExecutor.Invoke(static value => + { + string? cacheHome = value == "" ? null + : value == "absolute" ? Path.Combine(Path.GetTempPath(), "invocation-cache-" + Guid.NewGuid().ToString("N")) + : value; + Environment.SetEnvironmentVariable("XDG_CACHE_HOME", cacheHome); + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile); + string expectedDirectory = OperatingSystem.IsWindows() + ? Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "test") + : OperatingSystem.IsMacOS() + ? Path.Combine(home, "Library", "Caches", "test") + : Path.Combine(value == "absolute" ? cacheHome! : Path.Combine(home, ".cache"), "test"); + + InvocationCache? cache = InvocationCache.TryCreate(null, task.Log, "Test"); + + Assert.NotNull(cache); + Assert.Equal(Path.GetFullPath(expectedDirectory), cache.CacheDirectory); + }, setting ?? "").Dispose(); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void CacheRoundTrip(bool includeSymbols) + { + using var test = new CacheTestDirectory(); + var files = new Dictionary + { + ["app.dll"] = "assembly", + ["app.dll.config"] = "configuration", + [Path.Combine("fr", "app.resources.dll")] = "satellite", + ["Link.semaphore"] = "ordinary file" + }; + if (includeSymbols) + files.Add("app.pdb", "symbols"); + + foreach (var file in files) + test.WriteSource(file.Key, file.Value); + + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + + Directory.CreateDirectory(test.Output); + File.WriteAllText(Path.Combine(test.Output, "app.dll"), "old assembly"); + File.WriteAllText(Path.Combine(test.Output, "Link.semaphore"), "existing semaphore"); + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + + foreach (var file in files) + { + string path = Path.Combine(test.Output, file.Key); + Assert.Equal(file.Value, File.ReadAllText(path)); + Assert.Equal(CacheTestDirectory.Timestamp, File.GetLastWriteTimeUtc(path)); + } + Assert.Equal("ordinary file", File.ReadAllText(Path.Combine(test.Output, "Link.semaphore"))); + Assert.Equal(files.Count, Directory.GetFiles(Path.Combine(test.Entry, "outputs"), "*", SearchOption.AllDirectories).Length); + Assert.Empty(Directory.GetDirectories(test.Entries, "*.tmp")); + + File.WriteAllText(Path.Combine(test.Output, "app.dll"), "modified restored output"); + Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Entry, "outputs", "app.dll"))); + } + + [Fact] + public void CacheSupportsEmptyOutput() + { + using var test = new CacheTestDirectory(); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.True(Directory.Exists(test.Output)); + Assert.Empty(Directory.GetFileSystemEntries(test.Output)); + } + + [Fact] + public void CacheRecordsLastUsedOnStoreAndHit() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + DateTimeOffset beforeStore = DateTimeOffset.UtcNow; + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + string marker = Path.Combine(test.Entry, InvocationCacheEntry.LastUsedFileName); + Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), + beforeStore, DateTimeOffset.UtcNow); + File.WriteAllText(marker, DateTimeOffset.MinValue.ToString("O", CultureInfo.InvariantCulture)); + + DateTimeOffset beforeRestore = DateTimeOffset.UtcNow; + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + + Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), + beforeRestore, DateTimeOffset.UtcNow); + Assert.False(File.Exists(Path.Combine(test.Output, InvocationCacheEntry.LastUsedFileName))); + Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); + } + + [Fact] + public void CacheMarkerUpdateFailureDoesNotFailRestore() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + string marker = Path.Combine(test.Entry, InvocationCacheEntry.LastUsedFileName); + File.Delete(marker); + Directory.CreateDirectory(marker); + + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + + Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Output, "app.dll"))); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("last-used update failed") is true); + Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); + Assert.Empty(test.BuildEngine.Errors); + } + + [Fact] + public void FailedRestoreDoesNotRefreshUsage() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + string marker = Path.Combine(test.Entry, InvocationCacheEntry.LastUsedFileName); + string oldTimestamp = DateTimeOffset.MinValue.ToString("O", CultureInfo.InvariantCulture); + File.WriteAllText(marker, oldTimestamp); + File.WriteAllText(Path.Combine(test.Entry, "outputs", "app.dll"), "corrupt"); + + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + + Assert.Equal(oldTimestamp, File.ReadAllText(marker)); + } + + [Fact] + public void ParallelProcessesRefreshUsageWithoutCorruptingMarker() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + DateTimeOffset beforeRestore = DateTimeOffset.UtcNow; + using (StartReader(test, 0)) + using (StartReader(test, 1)) + { + Assert.True(SpinWait.SpinUntil(() => Directory.GetFiles(test.Root, "*.ready").Length == 2, TimeSpan.FromSeconds(30))); + File.WriteAllText(Path.Combine(test.Root, "start"), ""); + } + + string marker = Path.Combine(test.Entry, InvocationCacheEntry.LastUsedFileName); + Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), + beforeRestore, DateTimeOffset.UtcNow); + Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); + + static RemoteInvokeHandle StartReader(CacheTestDirectory test, int index) => + RemoteExecutor.Invoke(static (cacheDirectory, output, start) => + { + var engine = new CacheBuildEngine(); + var task = new CacheTask { BuildEngine = engine }; + var cache = new InvocationCache(cacheDirectory, task.Log, "Test"); + File.WriteAllText(output + ".ready", ""); + Assert.True(SpinWait.SpinUntil(() => File.Exists(start), TimeSpan.FromSeconds(30))); + for (int attempt = 0; attempt < 20; attempt++) + Assert.True(cache.TryRestore(CacheTestDirectory.Key, InvocationOutputs.FromDirectory(output), CacheTestDirectory.Timestamp)); + Assert.Empty(engine.Errors); + }, test.Cache.CacheDirectory, Path.Combine(test.Root, "reader-" + index), Path.Combine(test.Root, "start")); + } + + [Fact] + public void CacheDoesNotReplaceExistingEntry() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "first"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + test.WriteSource("app.dll", "second"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.Equal("first", File.ReadAllText(Path.Combine(test.Output, "app.dll"))); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("already exists") is true); + Assert.Empty(Directory.GetDirectories(test.Entries, "*.tmp")); + } + + [Theory] + [InlineData("missing-manifest")] + [InlineData("version")] + [InlineData("truncated")] + [InlineData("missing-file")] + [InlineData("corrupt-file")] + [InlineData("escaping-path")] + public void CacheRejectsInvalidEntryBeforeCopying(string damage) + { + using var test = new CacheTestDirectory(); + test.WriteSource("a.dll", "first"); + test.WriteSource("b.dll", "second"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + string manifest = Path.Combine(test.Entry, "manifest"); + string cachedFile = Path.Combine(test.Entry, "outputs", "b.dll"); + switch (damage) + { + case "missing-manifest": + File.Delete(manifest); + break; + case "version": + using (var writer = new BinaryWriter(File.OpenWrite(manifest))) + writer.Write(-1); + break; + case "truncated": + File.WriteAllBytes(manifest, new byte[] { 1 }); + break; + case "missing-file": + File.Delete(cachedFile); + break; + case "corrupt-file": + File.WriteAllText(cachedFile, "broken"); + break; + case "escaping-path": + using (var writer = new BinaryWriter(File.Create(manifest))) + { + writer.Write(InvocationCacheEntry.FormatVersion); + writer.Write(false); + writer.Write(1); + writer.Write("../escaped.dll"); + } + break; + } + if (File.Exists(manifest)) + File.WriteAllBytes(Path.Combine(test.Entry, "manifest.sha256"), SHA256.HashData(File.ReadAllBytes(manifest))); + + Directory.CreateDirectory(test.Output); + string existingOutput = Path.Combine(test.Output, "a.dll"); + File.WriteAllText(existingOutput, "unchanged"); + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.Equal("unchanged", File.ReadAllText(existingOutput)); + Assert.False(File.Exists(Path.Combine(test.Output, "b.dll"))); + Assert.False(File.Exists(Path.Combine(test.Root, "escaped.dll"))); + Assert.True(Directory.Exists(test.Entry)); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("restore failed") is true); + } + + [Fact] + public void CacheRestoreReplacesExistingDirectory() + { + using var test = new CacheTestDirectory(); + test.WriteSource("a.dll", "new assembly"); + test.WriteSource("b.dll", "other assembly"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + Directory.CreateDirectory(Path.Combine(test.Output, "b.dll")); + File.WriteAllText(Path.Combine(test.Output, "a.dll"), "old assembly"); + + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.Equal("new assembly", File.ReadAllText(Path.Combine(test.Output, "a.dll"))); + Assert.Equal("other assembly", File.ReadAllText(Path.Combine(test.Output, "b.dll"))); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void CacheStoreFailureIsBestEffort(bool failPublication) + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + if (failPublication) + { + Directory.CreateDirectory(test.Entries); + File.WriteAllText(test.Entry, "blocks publication"); + } + else + { + File.WriteAllText(test.Cache.CacheDirectory, "blocks cache creation"); + } + + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + + Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Source, "app.dll"))); + Assert.False(Directory.Exists(test.Entry)); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("store failed") is true); + if (failPublication) + { + Assert.Equal("blocks publication", File.ReadAllText(test.Entry)); + Assert.Empty(Directory.GetDirectories(test.Entries, "*.tmp")); + } + } + + [Fact] + public void CacheSkipsContendedEntryButNotOtherKeys() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + using var mutex = new Mutex(false, test.MutexName); + Assert.True(mutex.WaitOne(0)); + try + { + RunOnOtherThread(() => test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs)); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("another writer") is true); + + string otherKey = new string('b', 64); + RunOnOtherThread(() => test.Cache.Store(otherKey, test.SourceOutputs)); + Assert.True(test.Cache.TryRestore(otherKey, test.OutputOutputs, CacheTestDirectory.Timestamp)); + } + finally + { + mutex.ReleaseMutex(); + } + + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + Assert.True(Directory.Exists(test.Entry)); + } + + [Fact] + public void CacheReadsPublishedEntryWithoutWriterLock() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + using var mutex = new Mutex(false, test.MutexName); + Assert.True(mutex.WaitOne(0)); + try + { + RunOnOtherThread(() => + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp))); + } + finally + { + mutex.ReleaseMutex(); + } + } + + [Fact] + public void CacheSupportsConcurrentWritersWithDifferentOutputs() + { + using var test = new CacheTestDirectory(); + string contents = new string('a', 1024 * 1024); + System.Threading.Tasks.Parallel.For(0, 8, index => + { + string source = Path.Combine(test.Root, "source-" + index); + Directory.CreateDirectory(source); + File.WriteAllText(Path.Combine(source, "app.dll"), contents); + File.WriteAllText(Path.Combine(source, "app.pdb"), "symbols"); + var task = new CacheTask { BuildEngine = new CacheBuildEngine() }; + var cache = new InvocationCache(test.Cache.CacheDirectory, task.Log, "Test"); + + cache.Store(CacheTestDirectory.Key, InvocationOutputs.FromDirectory(source)); + }); + + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.Equal(contents, File.ReadAllText(Path.Combine(test.Output, "app.dll"))); + Assert.Equal("symbols", File.ReadAllText(Path.Combine(test.Output, "app.pdb"))); + Assert.Single(Directory.GetDirectories(test.Entries)); + } + + [Fact] + public void CacheIgnoresOtherWritersStagingDirectories() + { + using var test = new CacheTestDirectory(); + string stagingDirectory = test.Entry + ".orphan.tmp"; + Directory.CreateDirectory(stagingDirectory); + string marker = Path.Combine(stagingDirectory, "partial"); + File.WriteAllText(marker, "other attempt"); + + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + + Assert.Equal("other attempt", File.ReadAllText(marker)); + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void CacheRejectsOverlappingDirectories(bool cacheContainsOutput) + { + using var test = new CacheTestDirectory(); + string output = cacheContainsOutput ? Path.Combine(test.Cache.CacheDirectory, "output") : test.Root; + + Assert.Throws(() => test.Cache.Store(CacheTestDirectory.Key, InvocationOutputs.FromDirectory(output))); + Assert.Throws(() => test.Cache.TryRestore(CacheTestDirectory.Key, InvocationOutputs.FromDirectory(output), CacheTestDirectory.Timestamp)); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + } + + [Fact] + public void CacheRecoversAbandonedWriterLock() + { + using var test = new CacheTestDirectory(); + test.WriteSource("app.dll", "assembly"); + using var mutex = new Mutex(false, test.MutexName); + var owner = new Thread(() => mutex.WaitOne()); + owner.Start(); + owner.Join(); + + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + + Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("abandoned") is true); + } + + [Theory] + [InlineData("")] + [InlineData("../entry")] + [InlineData("not-a-hash")] + public void CacheRejectsInvalidInputHash(string inputHash) + { + using var test = new CacheTestDirectory(); + Assert.Throws(() => test.Cache.Store(inputHash, test.SourceOutputs)); + Assert.Throws(() => test.Cache.TryRestore(inputHash, test.OutputOutputs, CacheTestDirectory.Timestamp)); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public void SharedTaskAdapterGatesDescriptionAndUsesCapturedArguments(bool configured, bool eligible) + => RemoteExecutor.Invoke(static (enabled, supported) => + { + bool isConfigured = bool.Parse(enabled); + bool isEligible = bool.Parse(supported); + using var test = new CacheTestDirectory(); + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, enabled); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, test.Cache.CacheDirectory); + test.WriteSource("image.dll", "cached"); + var invocation = new InvocationDescription(InvocationInputs.FromFiles(), + new[] { "compiler path", "response contents", "command contents" }, test.OutputOutputs); + test.Cache.Store(InvocationCache.ComputeKey(invocation), test.SourceOutputs); + var events = new List(); + DateTime preparedAt = default; + + int result = InvocationCache.ExecuteCached(test.Log, s_cacheConfiguration, + describeInvocation: () => DescribeInvocation("compiler path", "response contents", "command contents"), + executeNormally: () => ExecuteNormally("compiler path", "response contents", "command contents")); + DateTime completedAt = DateTime.UtcNow; + + Assert.Equal(isConfigured && isEligible ? 0 : 42, result); + Assert.Equal(isConfigured + ? isEligible ? new[] { "describe" } : new[] { "describe", "execute" } + : new[] { "execute" }, events); + if (isConfigured && isEligible) + { + Assert.InRange(File.GetLastWriteTimeUtc(Path.Combine(test.Output, "image.dll")), preparedAt, completedAt); + Assert.Equal("cached", File.ReadAllText(Path.Combine(test.Output, "image.dll"))); + } + + InvocationDescription? DescribeInvocation(string path, string response, string command) + { + AssertToolArguments(path, response, command); + events.Add("describe"); + preparedAt = DateTime.UtcNow; + return isEligible ? invocation : null; + } + + int ExecuteNormally(string path, string response, string command) + { + AssertToolArguments(path, response, command); + events.Add("execute"); + return 42; + } + + static void AssertToolArguments(string path, string response, string command) + { + Assert.Equal("compiler path", path); + Assert.Equal("response contents", response); + Assert.Equal("command contents", command); + } + }, configured.ToString(), eligible.ToString()).Dispose(); + + [Fact] + public void SharedTaskAdapterPropagatesPreparationExceptions() => RemoteExecutor.Invoke(static () => + { + using var test = new CacheTestDirectory(); + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "true"); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, test.Cache.CacheDirectory); + InvalidOperationException exception = Assert.Throws(() => + InvocationCache.ExecuteCached(test.Log, s_cacheConfiguration, + describeInvocation: () => throw new InvalidOperationException("description"), + executeNormally: () => throw new InvalidOperationException("unexpected execution"))); + + Assert.Equal("description", exception.Message); + Assert.False(Directory.Exists(test.Entry)); + }).Dispose(); + + [Theory] + [InlineData("equal-directory")] + [InlineData("cache-in-directory")] + [InlineData("directory-in-cache")] + [InlineData("duplicate-files")] + [InlineData("nested-files")] + [InlineData("file-in-cache")] + [InlineData("cache-in-file")] + public void SharedTaskAdapterRejectsInvalidOutputsWithoutExecuting(string shape) => RemoteExecutor.Invoke(static value => + { + using var test = new CacheTestDirectory(); + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "true"); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, test.Cache.CacheDirectory); + test.WriteSource("image.dll", "cached"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + Directory.CreateDirectory(test.Output); + string stale = Path.Combine(test.Output, "stale.dll"); + File.WriteAllText(stale, "untouched"); + string image = Path.Combine(test.Output, "image.dll"); + InvocationOutputs outputs = value switch + { + "equal-directory" => InvocationOutputs.FromDirectory(test.Cache.CacheDirectory), + "cache-in-directory" => InvocationOutputs.FromDirectory(test.Root), + "directory-in-cache" => InvocationOutputs.FromDirectory(Path.Combine(test.Cache.CacheDirectory, "output")), + "duplicate-files" => InvocationOutputs.FromFiles(image, image), + "nested-files" => InvocationOutputs.FromFiles(image, Path.Combine(image, "child")), + "file-in-cache" => InvocationOutputs.FromFiles(Path.Combine(test.Cache.CacheDirectory, "image.dll")), + "cache-in-file" => InvocationOutputs.FromFiles(test.Root), + _ => throw new InvalidOperationException() + }; + var invocation = new InvocationDescription(InvocationInputs.FromFiles(), Array.Empty(), outputs); + int executions = 0; + + ArgumentException exception = Assert.Throws(() => + InvocationCache.ExecuteCached(test.Log, s_cacheConfiguration, + describeInvocation: () => invocation, + executeNormally: () => + { + executions++; + return 42; + })); + + Assert.Equal(nameof(outputs), exception.ParamName); + Assert.Equal(0, executions); + Assert.Equal("untouched", File.ReadAllText(stale)); + Assert.Equal("cached", File.ReadAllText(Path.Combine(test.Entry, "outputs", "image.dll"))); + }, shape).Dispose(); + + [Fact] + public void SharedTaskAdapterBypassesUnreadableInputWithoutCacheCleanup() => RemoteExecutor.Invoke(static () => + { + using var test = new CacheTestDirectory(); + Environment.SetEnvironmentVariable(EnabledEnvironmentVariable, "true"); + Environment.SetEnvironmentVariable(PathEnvironmentVariable, test.Cache.CacheDirectory); + Directory.CreateDirectory(test.Output); + string stale = Path.Combine(test.Output, "stale.dll"); + File.WriteAllText(stale, "untouched"); + var invocation = new InvocationDescription(InvocationInputs.FromFiles(test.Source), Array.Empty(), test.OutputOutputs); + int executions = 0; + + int result = InvocationCache.ExecuteCached(test.Log, s_cacheConfiguration, + describeInvocation: () => invocation, + executeNormally: () => + { + executions++; + Assert.Equal("untouched", File.ReadAllText(stale)); + return 42; + }); + + Assert.Equal(42, result); + Assert.Equal(1, executions); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("input identity could not be computed") is true); + }).Dispose(); + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + [InlineData(true, true)] + public void SharedCacheRestoresDeclaredOutputContract(bool ownedDirectory, bool includeOptional) + { + using var test = new CacheTestDirectory(); + InvocationCache cache = test.Cache; + test.WriteSource("image.dll", "image"); + if (includeOptional) + test.WriteSource(Path.Combine("symbols", "image.pdb"), "symbols"); + + cache.Store(CacheTestDirectory.Key, Contract(test.Source)); + Directory.CreateDirectory(Path.Combine(test.Output, "symbols")); + string stale = Path.Combine(test.Output, "unrelated.txt"); + File.WriteAllText(stale, "keep unless the whole directory is owned"); + File.WriteAllText(Path.Combine(test.Output, "symbols", "image.pdb"), "stale symbols"); + + Assert.True(cache.TryRestore(CacheTestDirectory.Key, Contract(test.Output), CacheTestDirectory.Timestamp)); + Assert.Equal("image", File.ReadAllText(Path.Combine(test.Output, "image.dll"))); + Assert.Equal(includeOptional, File.Exists(Path.Combine(test.Output, "symbols", "image.pdb"))); + if (includeOptional) + Assert.Equal("symbols", File.ReadAllText(Path.Combine(test.Output, "symbols", "image.pdb"))); + Assert.Equal(!ownedDirectory, File.Exists(stale)); + Assert.Equal(CacheTestDirectory.Timestamp, File.GetLastWriteTimeUtc(Path.Combine(test.Output, "image.dll"))); + + InvocationOutputs Contract(string directory) => ownedDirectory + ? InvocationOutputs.FromDirectory(directory) + : InvocationOutputs.FromFiles( + Path.Combine(directory, "image.dll"), + Path.Combine(directory, "symbols", "image.pdb")); + } + + [Fact] + public void SharedCacheRecordsAbsentDeclaredFiles() + { + using var test = new CacheTestDirectory(); + InvocationCache cache = test.Cache; + InvocationOutputs outputs = InvocationOutputs.FromFiles( + Path.Combine(test.Output, "missing.dll")); + cache.Store(CacheTestDirectory.Key, outputs); + Assert.True(Directory.Exists(test.Entry)); + Directory.CreateDirectory(test.Output); + File.WriteAllText(Path.Combine(test.Output, "missing.dll"), "stale"); + Assert.True(cache.TryRestore(CacheTestDirectory.Key, outputs, CacheTestDirectory.Timestamp)); + Assert.False(File.Exists(Path.Combine(test.Output, "missing.dll"))); + Assert.Empty(Directory.GetDirectories(Path.Combine(cache.CacheDirectory, InvocationCacheEntry.VersionDirectory), "*.tmp")); + } + + [Theory] + [InlineData("content")] + [InlineData("absence")] + [InlineData("context")] + [InlineData("output")] + [InlineData("input-order")] + public void SharedFingerprintIncludesDeclaredInputsContextAndOutputs(string change) + { + using var test = new CacheTestDirectory(); + test.WriteSource("first.dll", "first"); + test.WriteSource("second.dll", "second"); + string first = Path.Combine(test.Source, "first.dll"); + string second = Path.Combine(test.Source, "second.dll"); + string missing = Path.Combine(test.Source, "missing.dll"); + string[] inputs = new[] { first, second, missing }; + string[] context = new[] { "arguments" }; + InvocationOutputs outputs = test.OutputOutputs; + string before = InvocationCache.ComputeKey(new InvocationDescription(InvocationInputs.FromFiles(inputs), context, outputs)); + + switch (change) + { + case "content": File.WriteAllText(first, "changed"); break; + case "absence": File.WriteAllText(missing, "now present"); break; + case "context": context = new[] { "changed arguments" }; break; + case "output": outputs = InvocationOutputs.FromFiles(Path.Combine(test.Output, "other.dll")); break; + case "input-order": inputs = new[] { second, first, missing }; break; + } + + Assert.NotEqual(before, InvocationCache.ComputeKey(new InvocationDescription(InvocationInputs.FromFiles(inputs), context, outputs))); + } + + [Fact] + public void SharedFingerprintDoesNotTreatUnreadableInputsAsAbsent() + { + using var test = new CacheTestDirectory(); + var invocation = new InvocationDescription(InvocationInputs.FromFiles(test.Source), Array.Empty(), test.OutputOutputs); + Assert.False(InvocationCache.TryComputeKey(test.Log, "Test", invocation, out string key)); + Assert.Equal(string.Empty, key); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("input identity could not be computed") is true); + } + + [Fact] + public void SharedExecutionRecordsAbsentOutputsAfterSuccess() + { + using var test = new CacheTestDirectory(); + Directory.CreateDirectory(test.Output); + string image = Path.Combine(test.Output, "image.dll"); + string symbols = Path.Combine(test.Output, "image.pdb"); + File.WriteAllText(image, "stale image"); + File.WriteAllText(symbols, "stale symbols"); + var request = new InvocationCacheRequest(CacheTestDirectory.Key, InvocationOutputs.FromFiles(image, symbols)); + int executions = 0; + + int Run() => InvocationCache.ExecutePrepared(test.Cache, CacheTestDirectory.Timestamp, request, () => + { + executions++; + Assert.Equal("stale image", File.ReadAllText(image)); + Assert.Equal("stale symbols", File.ReadAllText(symbols)); + File.Delete(image); + File.Delete(symbols); + return 0; + }); + + Assert.Equal(0, Run()); + Assert.True(Directory.Exists(test.Entry)); + File.WriteAllText(image, "new stale image"); + File.WriteAllText(symbols, "new stale symbols"); + Assert.Equal(0, Run()); + Assert.Equal(1, executions); + Assert.False(File.Exists(image)); + Assert.False(File.Exists(symbols)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void SharedCacheUsesOneManifestFormat(bool ownedDirectory) + { + using var test = new CacheTestDirectory(); + test.WriteSource("image.dll", "image"); + InvocationOutputs source = ownedDirectory ? test.SourceOutputs + : InvocationOutputs.FromFiles(Path.Combine(test.Source, "image.dll")); + test.Cache.Store(CacheTestDirectory.Key, source); + + Assert.Equal("v1", InvocationCacheEntry.VersionDirectory); + byte[] manifestBytes = File.ReadAllBytes(Path.Combine(test.Entry, "manifest")); + Assert.Equal(SHA256.HashData(manifestBytes), File.ReadAllBytes(Path.Combine(test.Entry, "manifest.sha256"))); + using var manifest = new BinaryReader(new MemoryStream(manifestBytes)); + Assert.Equal(1, manifest.ReadInt32()); + Assert.Equal(!ownedDirectory, manifest.ReadBoolean()); + Assert.Equal(1, manifest.ReadInt32()); + Assert.Equal(ownedDirectory ? "image.dll" : "0", manifest.ReadString()); + Assert.True(manifest.ReadBoolean()); + byte[] image = File.ReadAllBytes(Path.Combine(test.Source, "image.dll")); + Assert.Equal(image.Length, manifest.ReadInt64()); + Assert.Equal(SHA256.HashData(image), manifest.ReadBytes(32)); + Assert.Equal(manifest.BaseStream.Length, manifest.BaseStream.Position); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void SharedCacheRejectsDifferentOutputContract(bool ownedDirectory) + { + using var test = new CacheTestDirectory(); + test.WriteSource("0", "cached"); + InvocationOutputs source = ownedDirectory ? test.SourceOutputs + : InvocationOutputs.FromFiles(Path.Combine(test.Source, "0")); + test.Cache.Store(CacheTestDirectory.Key, source); + Directory.CreateDirectory(test.Output); + string image = Path.Combine(test.Output, "0"); + File.WriteAllText(image, "untouched"); + InvocationOutputs destination = ownedDirectory + ? InvocationOutputs.FromFiles(image) + : test.OutputOutputs; + + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, destination, CacheTestDirectory.Timestamp)); + + Assert.Equal("untouched", File.ReadAllText(image)); + } + + [Theory] + [InlineData(false, "missing-checksum")] + [InlineData(false, "corrupt-checksum")] + [InlineData(false, "corrupt-manifest")] + [InlineData(true, "missing-checksum")] + [InlineData(true, "corrupt-checksum")] + [InlineData(true, "corrupt-manifest")] + public void SharedCacheValidatesManifestIntegrityBeforeModifyingOutputs(bool ownedDirectory, string damage) + { + using var test = new CacheTestDirectory(); + test.WriteSource("image.dll", "cached"); + InvocationOutputs source = ownedDirectory ? test.SourceOutputs + : InvocationOutputs.FromFiles(Path.Combine(test.Source, "image.dll")); + test.Cache.Store(CacheTestDirectory.Key, source); + string checksum = Path.Combine(test.Entry, "manifest.sha256"); + switch (damage) + { + case "missing-checksum": + File.Delete(checksum); + break; + case "corrupt-checksum": + File.WriteAllBytes(checksum, new byte[32]); + break; + case "corrupt-manifest": + File.WriteAllText(Path.Combine(test.Entry, "manifest"), "corrupt"); + break; + } + Directory.CreateDirectory(test.Output); + string image = Path.Combine(test.Output, "image.dll"); + File.WriteAllText(image, "untouched"); + InvocationOutputs destination = ownedDirectory ? test.OutputOutputs + : InvocationOutputs.FromFiles(image); + + Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, destination, CacheTestDirectory.Timestamp)); + + Assert.Equal("untouched", File.ReadAllText(image)); + } + + [Theory] + [InlineData("duplicate")] + [InlineData("nested")] + [InlineData("cache-child")] + [InlineData("cache-parent")] + public void SharedCacheRejectsOverlappingFileDeclarations(string overlap) + { + using var test = new CacheTestDirectory(); + InvocationCache cache = test.Cache; + string first = Path.Combine(test.Output, "image.dll"); + string second = overlap switch + { + "duplicate" => first, + "nested" => Path.Combine(first, "child"), + "cache-child" => Path.Combine(cache.CacheDirectory, "output.dll"), + "cache-parent" => test.Root, + _ => throw new InvalidOperationException() + }; + InvocationOutputs outputs = InvocationOutputs.FromFiles( + first, second); + Assert.Throws(() => cache.Store(CacheTestDirectory.Key, outputs)); + Assert.Throws(() => cache.TryRestore(CacheTestDirectory.Key, outputs, CacheTestDirectory.Timestamp)); + Assert.False(Directory.Exists(cache.CacheDirectory)); + } + + [Theory] + [InlineData(false, false)] + [InlineData(false, true)] + [InlineData(true, false)] + public void SharedExecutionBypassesWithoutSkippingTaskPreparation(bool configured, bool eligible) + { + using var test = new CacheTestDirectory(); + Directory.CreateDirectory(test.Output); + string stale = Path.Combine(test.Output, "stale.dll"); + File.WriteAllText(stale, "stale"); + int executions = 0; + + int exitCode = InvocationCache.ExecutePrepared(configured ? test.Cache : null, CacheTestDirectory.Timestamp, + eligible ? new InvocationCacheRequest(CacheTestDirectory.Key, test.OutputOutputs) : null, + () => + { + executions++; + Directory.Delete(test.Output, recursive: true); + Directory.CreateDirectory(test.Output); + return 42; + }); + + Assert.Equal(42, exitCode); + Assert.Equal(1, executions); + Assert.False(File.Exists(stale)); + Assert.False(Directory.Exists(test.Cache.CacheDirectory)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void SharedExecutionStoresSuccessfulResultsAndRestoresWithoutExecuting(bool ownedDirectory) + { + using var test = new CacheTestDirectory(); + string image = Path.Combine(test.Output, "image.dll"); + string symbols = Path.Combine(test.Output, "image.pdb"); + InvocationOutputs outputs = ownedDirectory ? test.OutputOutputs + : InvocationOutputs.FromFiles(image, symbols); + var request = new InvocationCacheRequest(CacheTestDirectory.Key, outputs); + var events = new List(); + Directory.CreateDirectory(test.Output); + File.WriteAllText(image, "stale image"); + File.WriteAllText(symbols, "stale symbols"); + + int Run() => InvocationCache.ExecutePrepared(test.Cache, CacheTestDirectory.Timestamp, request, + () => + { + events.Add("execute"); + Assert.Equal("stale image", File.ReadAllText(image)); + Assert.Equal("stale symbols", File.ReadAllText(symbols)); + if (ownedDirectory) + { + Directory.Delete(test.Output, recursive: true); + } + else + { + File.Delete(image); + File.Delete(symbols); + } + Directory.CreateDirectory(test.Output); + File.WriteAllText(image, "computed"); + return 0; + }); + + Assert.Equal(0, Run()); + Assert.Equal(new[] { "execute" }, events); + Assert.True(Directory.Exists(test.Entry)); + events.Clear(); + File.WriteAllText(image, "outdated"); + File.WriteAllText(symbols, "stale symbols"); + string neighbor = Path.Combine(test.Output, "unrelated.txt"); + File.WriteAllText(neighbor, "neighbor"); + + Assert.Equal(0, Run()); + + Assert.Empty(events); + Assert.Equal("computed", File.ReadAllText(image)); + Assert.False(File.Exists(symbols)); + Assert.Equal(!ownedDirectory, File.Exists(neighbor)); + } + + [Theory] + [InlineData(0, true)] + [InlineData(1, false)] + [InlineData(-1, false)] + [InlineData(1, true)] + public void SharedExecutionDoesNotStoreFailedOrErrorLoggedResults(int exitCode, bool logError) + { + using var test = new CacheTestDirectory(); + int executions = 0; + int result = InvocationCache.ExecutePrepared(test.Cache, CacheTestDirectory.Timestamp, + new InvocationCacheRequest(CacheTestDirectory.Key, test.OutputOutputs), + () => + { + executions++; + Directory.CreateDirectory(test.Output); + File.WriteAllText(Path.Combine(test.Output, "image.dll"), "produced"); + if (logError) + test.Log.LogError("Expected compiler error."); + return exitCode; + }); + + Assert.Equal(exitCode, result); + Assert.Equal(1, executions); + Assert.False(Directory.Exists(test.Entry)); + } + + [Fact] + public void SharedExecutionPropagatesTaskExceptions() + { + using var test = new CacheTestDirectory(); + int executions = 0; + InvalidOperationException exception = Assert.Throws(() => + InvocationCache.ExecutePrepared(test.Cache, CacheTestDirectory.Timestamp, + new InvocationCacheRequest(CacheTestDirectory.Key, test.OutputOutputs), + () => + { + executions++; + throw new InvalidOperationException("execute"); + })); + + Assert.Equal("execute", exception.Message); + Assert.Equal(1, executions); + Assert.False(Directory.Exists(test.Entry)); + } + + [Fact] + public void SharedExecutionFallsBackAfterRestoreFailure() + { + using var test = new CacheTestDirectory(); + test.WriteSource("image.dll", "cached"); + test.Cache.Store(CacheTestDirectory.Key, test.SourceOutputs); + File.WriteAllText(Path.Combine(test.Entry, "outputs", "image.dll"), "corrupt"); + Directory.CreateDirectory(test.Output); + string stale = Path.Combine(test.Output, "stale.dll"); + File.WriteAllText(stale, "stale"); + int executions = 0; + + Assert.Equal(0, InvocationCache.ExecutePrepared(test.Cache, CacheTestDirectory.Timestamp, + new InvocationCacheRequest(CacheTestDirectory.Key, test.OutputOutputs), + () => + { + executions++; + Assert.Equal("stale", File.ReadAllText(stale)); + Directory.Delete(test.Output, recursive: true); + Directory.CreateDirectory(test.Output); + File.WriteAllText(Path.Combine(test.Output, "image.dll"), "recomputed"); + return 0; + })); + + Assert.Equal(1, executions); + Assert.False(File.Exists(stale)); + Assert.Equal("recomputed", File.ReadAllText(Path.Combine(test.Output, "image.dll"))); + Assert.Contains(test.BuildEngine.Messages, message => message.Message?.Contains("restore failed") is true); + } + + private static void RunOnOtherThread(Action action) + { + ExceptionDispatchInfo? exception = null; + var thread = new Thread(() => + { + try + { + action(); + } + catch (Exception ex) + { + exception = ExceptionDispatchInfo.Capture(ex); + } + }); + thread.Start(); + thread.Join(); + exception?.Throw(); + } + + private sealed class CacheTestDirectory : IDisposable + { + internal static string Key { get; } = new string('a', 64); + internal static DateTime Timestamp { get; } = new DateTime(2025, 1, 2, 3, 4, 5, DateTimeKind.Utc); + internal string Root { get; } = Path.Combine(Path.GetTempPath(), "invocation-cache-tests-" + Guid.NewGuid().ToString("N")); + internal string Source => Path.Combine(Root, "source"); + internal string Output => Path.Combine(Root, "output"); + internal InvocationOutputs SourceOutputs { get; } + internal InvocationOutputs OutputOutputs { get; } + internal string Entries => Path.Combine(Cache.CacheDirectory, InvocationCacheEntry.VersionDirectory); + internal string Entry => Path.Combine(Entries, Key); + internal CacheBuildEngine BuildEngine { get; } = new(); + internal TaskLoggingHelper Log { get; } + internal InvocationCache Cache { get; } + internal string MutexName => Assert.IsType( + typeof(InvocationCache).GetMethod("GetWriterMutexName", BindingFlags.NonPublic | BindingFlags.Instance)!.Invoke(Cache, new object[] { Key })); + + internal CacheTestDirectory() + { + Directory.CreateDirectory(Source); + var task = new CacheTask { BuildEngine = BuildEngine }; + Log = task.Log; + Cache = new InvocationCache(Path.Combine(Root, "cache"), Log, "Test"); + SourceOutputs = InvocationOutputs.FromDirectory(Source); + OutputOutputs = InvocationOutputs.FromDirectory(Output); + } + + internal void WriteSource(string relativePath, string contents) + { + string path = Path.Combine(Source, relativePath); + Directory.CreateDirectory(Path.GetDirectoryName(path)!); + File.WriteAllText(path, contents); + } + + public void Dispose() => Directory.Delete(Root, recursive: true); + } + + private sealed class CacheTask : Microsoft.Build.Utilities.Task + { + public override bool Execute() => true; + } + + private sealed class CacheBuildEngine : IBuildEngine + { + internal List Errors { get; } = new(); + internal List Messages { get; } = new(); + + public void LogErrorEvent(BuildErrorEventArgs e) => Errors.Add(e); + public void LogWarningEvent(BuildWarningEventArgs e) { } + public void LogMessageEvent(BuildMessageEventArgs e) => Messages.Add(e); + public void LogCustomEvent(CustomBuildEventArgs e) { } + public bool BuildProjectFile(string projectFileName, string[] targetNames, IDictionary globalProperties, IDictionary targetOutputs) => false; + public bool ContinueOnError => false; + public int LineNumberOfTaskNode => 0; + public int ColumnNumberOfTaskNode => 0; + public string ProjectFileOfTaskNode => string.Empty; + } +} diff --git a/src/tasks/Common/InvocationCache/InvocationCache.Configuration.cs b/src/tasks/Common/InvocationCache/InvocationCache.Configuration.cs new file mode 100644 index 00000000000000..c0a19762df7e07 --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationCache.Configuration.cs @@ -0,0 +1,54 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.IO; +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed partial class InvocationCache +{ + internal static InvocationCache? TryCreateFromEnvironment(TaskLoggingHelper log, InvocationCacheConfiguration configuration) + { + string? setting = Environment.GetEnvironmentVariable(configuration.EnabledEnvironmentVariable); + if (string.IsNullOrEmpty(setting)) + return null; + + if (!bool.TryParse(setting, out bool enabled)) + { + log.LogMessage(MessageImportance.Low, $"{configuration.ToolName} caching is disabled because {configuration.EnabledEnvironmentVariable} must be 'true' or 'false'."); + return null; + } + + return enabled ? TryCreate(Environment.GetEnvironmentVariable(configuration.PathEnvironmentVariable), log, configuration.ToolName) : null; + } + + internal static InvocationCache? TryCreate(string? cacheDirectory, TaskLoggingHelper log, string toolName) + { + if (string.IsNullOrEmpty(cacheDirectory)) + cacheDirectory = InvocationCacheConfiguration.GetDefaultCacheDirectory(toolName); + + if (cacheDirectory is null) + { + log.LogMessage(MessageImportance.Low, $"{toolName} caching is disabled because no default cache directory is available."); + return null; + } + + try + { + string fullPath = Path.GetFullPath(cacheDirectory); + if (File.Exists(fullPath)) + throw new IOException($"Cache location is a file: '{fullPath}'."); + return new InvocationCache(fullPath, log, toolName); + } + catch (Exception ex) when (ex is ArgumentException or NotSupportedException or IOException or UnauthorizedAccessException) + { + log.LogMessage(MessageImportance.Low, $"{toolName} caching is disabled because the cache directory '{cacheDirectory}' could not be resolved: {ex.Message}"); + return null; + } + } +} diff --git a/src/tasks/Common/InvocationCache/InvocationCache.Identity.cs b/src/tasks/Common/InvocationCache/InvocationCache.Identity.cs new file mode 100644 index 00000000000000..3714a2054b2f21 --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationCache.Identity.cs @@ -0,0 +1,81 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.Globalization; +using System.IO; +using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text; +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed partial class InvocationCache +{ + internal static bool TryComputeKey(TaskLoggingHelper log, string toolName, InvocationDescription invocation, out string key) + { + key = string.Empty; + try + { + key = ComputeKey(invocation); + return true; + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or ArgumentException or NotSupportedException) + { + log.LogMessage(MessageImportance.Low, $"{toolName} cache bypassed: input identity could not be computed: {ex.Message}"); + return false; + } + } + + internal static string ComputeKey(InvocationDescription invocation) + { + using var sha256 = SHA256.Create(); + using var data = new MemoryStream(); + using var writer = new BinaryWriter(data, Encoding.UTF8, leaveOpen: true); + writer.Write("invocation-cache-key-v1"); + writer.Write(Environment.CurrentDirectory); + writer.Write(RuntimeInformation.OSDescription); + writer.Write(RuntimeInformation.ProcessArchitecture.ToString()); + writer.Write(CultureInfo.CurrentCulture.Name); + writer.Write(CultureInfo.CurrentUICulture.Name); + writer.Write(invocation.Context.Length); + foreach (string value in invocation.Context) + writer.Write(value); + writer.Write(invocation.Outputs.DirectoryPath is not null); + if (invocation.Outputs.DirectoryPath is not null) + writer.Write(invocation.Outputs.DirectoryPath); + writer.Write(invocation.Outputs.Files.Length); + foreach (string file in invocation.Outputs.Files) + writer.Write(file); + writer.Write(invocation.Inputs.Files.Length); + foreach (string input in invocation.Inputs.Files) + { + writer.Write(input); + try + { + using FileStream file = File.OpenRead(input); + writer.Write(true); + writer.Write(sha256.ComputeHash(file)); + } + catch (FileNotFoundException) + { + writer.Write(false); + } + catch (DirectoryNotFoundException) + { + writer.Write(false); + } + } + writer.Flush(); + data.Position = 0; +#if NET + return Convert.ToHexStringLower(sha256.ComputeHash(data)); +#else + return BitConverter.ToString(sha256.ComputeHash(data)).Replace("-", "").ToLowerInvariant(); +#endif + } +} diff --git a/src/tasks/Common/InvocationCache/InvocationCache.Outputs.cs b/src/tasks/Common/InvocationCache/InvocationCache.Outputs.cs new file mode 100644 index 00000000000000..cc888290f2b87a --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationCache.Outputs.cs @@ -0,0 +1,89 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Runtime.InteropServices; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed partial class InvocationCache +{ + internal void ValidateOutputs(InvocationOutputs outputs) + { + StringComparison comparison = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) + ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal; + string cachePrefix = CacheDirectory.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; + if (outputs.DirectoryPath is not null) + { + string outputPrefix = outputs.DirectoryPath.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; + if (cachePrefix.StartsWith(outputPrefix, comparison) || outputPrefix.StartsWith(cachePrefix, comparison)) + throw new ArgumentException("The cache and output directories must not contain each other.", nameof(outputs)); + } + else + { + var paths = new HashSet(RuntimeInformation.IsOSPlatform(OSPlatform.Windows) + ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal); + foreach (string file in outputs.Files) + { + string filePrefix = file.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; + foreach (string existing in paths) + { + string existingPrefix = existing.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; + if (existing.StartsWith(filePrefix, comparison) || file.StartsWith(existingPrefix, comparison)) + throw new ArgumentException("Declared output files must not contain each other.", nameof(outputs)); + } + if (!paths.Add(file) || file.StartsWith(cachePrefix, comparison) || + cachePrefix.StartsWith(filePrefix, comparison)) + throw new ArgumentException("Cache outputs must be distinct files outside the cache.", nameof(outputs)); + } + } + } + + private static List GetFileNames(InvocationOutputs outputs) + { + var files = new List(); + if (outputs.DirectoryPath is not null) + { + Collect(outputs.DirectoryPath, ""); + files.Sort(StringComparer.Ordinal); + } + else + { + for (int i = 0; i < outputs.Files.Length; i++) + files.Add(i.ToString(CultureInfo.InvariantCulture)); + } + return files; + + void Collect(string directory, string relativeDirectory) + { + foreach (string path in Directory.EnumerateFileSystemEntries(directory)) + { + string relativePath = Path.Combine(relativeDirectory, Path.GetFileName(path)); + if (Directory.Exists(path)) + Collect(path, relativePath); + else + files.Add(relativePath); + } + } + } + + private static string GetOutputPath(InvocationOutputs outputs, string name, int index) => + outputs.DirectoryPath is not null ? Path.Combine(outputs.DirectoryPath, name) : outputs.Files[index]; + + private static void DeleteOutputFile(string path) + { + try + { + File.Delete(path); + } + catch (DirectoryNotFoundException) + { + // A missing parent already satisfies the declared file's absence. + } + } +} diff --git a/src/tasks/Common/InvocationCache/InvocationCache.cs b/src/tasks/Common/InvocationCache/InvocationCache.cs new file mode 100644 index 00000000000000..92d119a2853829 --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationCache.cs @@ -0,0 +1,355 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using System.Runtime.InteropServices; +using System.Security.Cryptography; +using System.Text; +using System.Threading; +using Microsoft.Build.Framework; +using Microsoft.Build.Utilities; + +namespace Microsoft.NET.Build.Tasks; + +internal readonly struct InvocationCacheRequest +{ + internal string Key { get; } + internal InvocationOutputs Outputs { get; } + + internal InvocationCacheRequest(string key, InvocationOutputs outputs) + { + Key = key; + Outputs = outputs; + } +} + +internal sealed partial class InvocationCache +{ + private readonly TaskLoggingHelper _log; + private readonly string _toolName; + private readonly MessageImportance _importance; + + internal string CacheDirectory { get; } + + internal InvocationCache(string cacheDirectory, TaskLoggingHelper log, string toolName, + MessageImportance importance = MessageImportance.Low) + { + CacheDirectory = cacheDirectory; + _log = log; + _toolName = toolName; + _importance = importance; + } + + internal static int ExecuteCached(TaskLoggingHelper log, InvocationCacheConfiguration configuration, + Func describeInvocation, Func executeNormally) + => TryExecuteCached(log, configuration, describeInvocation, executeNormally, out int exitCode) + ? exitCode : executeNormally(); + + private static bool TryExecuteCached(TaskLoggingHelper log, InvocationCacheConfiguration configuration, + Func describeInvocation, Func executeNormally, out int exitCode) + { + exitCode = 0; + InvocationCache? cache = TryCreateFromEnvironment(log, configuration); + if (cache is null) + return false; + + InvocationDescription? invocation = describeInvocation(); + if (invocation is null) + return false; + + cache.ValidateOutputs(invocation.Outputs); + + DateTime outputTimestampUtc = DateTime.UtcNow; + if (!TryComputeKey(log, configuration.ToolName, invocation, out string key)) + return false; + + exitCode = ExecutePrepared(cache, outputTimestampUtc, + new InvocationCacheRequest(key, invocation.Outputs), executeNormally); + return true; + } + + internal static int ExecutePrepared(InvocationCache? cache, DateTime outputTimestampUtc, + InvocationCacheRequest? request, + Func execute) + { + if (cache is null || request is null) + return execute(); + + string key = request.Value.Key; + InvocationOutputs outputs = request.Value.Outputs; + if (cache.TryRestore(key, outputs, outputTimestampUtc)) + return 0; + + int exitCode = execute(); + if (exitCode == 0 && !cache._log.HasLoggedErrors) + cache.Store(key, outputs); + + return exitCode; + } + + internal bool TryRestore(string inputHash, InvocationOutputs outputs, DateTime outputTimestampUtc) + { + string entryDirectory = GetEntryDirectory(inputHash); + ValidateOutputs(outputs); + if (outputTimestampUtc.Kind != DateTimeKind.Utc) + throw new ArgumentException("The output timestamp must be UTC.", nameof(outputTimestampUtc)); + + if (!Directory.Exists(entryDirectory)) + { + _log.LogMessage(_importance, $"{_toolName} cache miss: {inputHash}"); + return false; + } + + try + { + List<(string Name, bool Present)> files = ReadManifest(entryDirectory, outputs); + + // Check the entire entry before modifying outputs. Copies can still fail if the cache is wiped concurrently. + if (outputs.DirectoryPath is not null) + { + if (Directory.Exists(outputs.DirectoryPath)) + Directory.Delete(outputs.DirectoryPath, recursive: true); + Directory.CreateDirectory(outputs.DirectoryPath); + } + for (int i = 0; i < files.Count; i++) + { + string destination = GetOutputPath(outputs, files[i].Name, i); + if (files[i].Present) + { + Directory.CreateDirectory(Path.GetDirectoryName(destination)!); + File.Copy(Path.Combine(entryDirectory, "outputs", files[i].Name), destination, overwrite: true); + File.SetLastWriteTimeUtc(destination, outputTimestampUtc); + } + else + { + DeleteOutputFile(destination); + } + } + + UpdateLastUsed(entryDirectory); + _log.LogMessage(_importance, $"{_toolName} cache hit: {inputHash}"); + return true; + } + catch (Exception ex) when (IsCacheFailure(ex)) + { + _log.LogMessage(_importance, $"{_toolName} cache restore failed for '{inputHash}'; running {_toolName} normally: {ex.Message}"); + return false; + } + } + + internal void Store(string inputHash, InvocationOutputs outputs) + { + string entryDirectory = GetEntryDirectory(inputHash); + ValidateOutputs(outputs); + string? stagingDirectory = null; + + try + { + using var mutex = new Mutex(initiallyOwned: false, GetWriterMutexName(inputHash)); + bool acquired; + try + { + acquired = mutex.WaitOne(0); + } + catch (AbandonedMutexException) + { + acquired = true; + _log.LogMessage(_importance, $"{_toolName} cache writer lock was abandoned: {inputHash}"); + } + + if (!acquired) + { + _log.LogMessage(_importance, $"{_toolName} cache store skipped; another writer owns '{inputHash}'."); + return; + } + + try + { + if (Directory.Exists(entryDirectory)) + { + _log.LogMessage(_importance, $"{_toolName} cache store skipped; entry '{inputHash}' already exists."); + return; + } + + List files = GetFileNames(outputs); + bool fileSet = outputs.DirectoryPath is null; + + string parentDirectory = Path.GetDirectoryName(entryDirectory)!; + Directory.CreateDirectory(parentDirectory); + stagingDirectory = Path.Combine(parentDirectory, inputHash + "." + Guid.NewGuid().ToString("N") + ".tmp"); + Directory.CreateDirectory(stagingDirectory); + + using (var manifest = new BinaryWriter(File.Create(Path.Combine(stagingDirectory, "manifest")))) + { + manifest.Write(InvocationCacheEntry.FormatVersion); + manifest.Write(fileSet); + manifest.Write(files.Count); + for (int i = 0; i < files.Count; i++) + { + string relativePath = files[i]; + string cachedFile = Path.Combine(stagingDirectory, "outputs", relativePath); + Directory.CreateDirectory(Path.GetDirectoryName(cachedFile)!); + bool present = true; + try + { + File.Copy(GetOutputPath(outputs, relativePath, i), cachedFile); + } + catch (Exception ex) when (fileSet && ex is FileNotFoundException or DirectoryNotFoundException) + { + present = false; + } + manifest.Write(relativePath); + manifest.Write(present); + if (present) + { + manifest.Write(new FileInfo(cachedFile).Length); + manifest.Write(HashFile(cachedFile)); + } + } + } + File.WriteAllBytes(Path.Combine(stagingDirectory, "manifest.sha256"), HashFile(Path.Combine(stagingDirectory, "manifest"))); + + InvocationCacheEntry.WriteLastUsed(Path.Combine(stagingDirectory, InvocationCacheEntry.LastUsedFileName)); + Directory.Move(stagingDirectory, entryDirectory); + stagingDirectory = null; + _log.LogMessage(_importance, $"{_toolName} cache stored: {inputHash}"); + } + finally + { + mutex.ReleaseMutex(); + } + } + catch (Exception ex) when (IsCacheFailure(ex)) + { + _log.LogMessage(_importance, $"{_toolName} cache store failed for '{inputHash}': {ex.Message}"); + } + finally + { + if (stagingDirectory is not null) + { + try + { + if (Directory.Exists(stagingDirectory)) + Directory.Delete(stagingDirectory, recursive: true); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + _log.LogMessage(_importance, $"{_toolName} cache staging cleanup failed for '{stagingDirectory}': {ex.Message}"); + } + } + } + } + + private void UpdateLastUsed(string entryDirectory) + { + try + { + InvocationCacheEntry.WriteLastUsed(Path.Combine(entryDirectory, InvocationCacheEntry.LastUsedFileName)); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) + { + _log.LogMessage(_importance, $"{_toolName} cache last-used update failed for '{entryDirectory}': {ex.Message}"); + } + } + + private string GetEntryDirectory(string inputHash) + { + if (!InvocationCacheEntry.IsEntryName(inputHash)) + throw new ArgumentException("The input hash must be a lowercase SHA-256 hex string.", nameof(inputHash)); + + return Path.Combine(CacheDirectory, InvocationCacheEntry.VersionDirectory, inputHash); + } + + private string GetWriterMutexName(string inputHash) + { + string directory = CacheDirectory.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + directory = directory.ToUpperInvariant(); + +#if NET + byte[] hash = SHA256.HashData(Encoding.UTF8.GetBytes(directory + "\n" + inputHash)); + return @"Global\" + _toolName + "Cache-" + Convert.ToHexString(hash); +#else + using var sha256 = SHA256.Create(); + byte[] hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(directory + "\n" + inputHash)); + return @"Global\" + _toolName + "Cache-" + BitConverter.ToString(hash).Replace("-", ""); +#endif + } + + private List<(string Name, bool Present)> ReadManifest(string entryDirectory, InvocationOutputs outputs) + { + bool fileSet = outputs.DirectoryPath is null; + if (!File.ReadAllBytes(Path.Combine(entryDirectory, "manifest.sha256")).SequenceEqual(HashFile(Path.Combine(entryDirectory, "manifest")))) + throw new InvalidDataException("Cached manifest checksum mismatch."); + + using var manifest = new BinaryReader(File.OpenRead(Path.Combine(entryDirectory, "manifest"))); + if (manifest.ReadInt32() != InvocationCacheEntry.FormatVersion) + throw new InvalidDataException($"Unsupported {_toolName} cache format."); + if (manifest.ReadBoolean() != fileSet) + throw new InvalidDataException($"Cached {_toolName} output contract does not match the invocation."); + + int fileCount = manifest.ReadInt32(); + if (fileCount < 0 || fileCount > manifest.BaseStream.Length - manifest.BaseStream.Position) + throw new InvalidDataException($"Invalid {_toolName} cache file count."); + if (fileSet && fileCount != outputs.Files.Length) + throw new InvalidDataException($"Invalid {_toolName} cache output inventory."); + + var files = new List<(string Name, bool Present)>(); + var paths = new HashSet(RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal); + string outputsDirectory = Path.Combine(entryDirectory, "outputs"); + for (int i = 0; i < fileCount; i++) + { + string relativePath = manifest.ReadString(); + ValidateRelativePath(relativePath); + if (!paths.Add(relativePath) || (fileSet && relativePath != i.ToString(CultureInfo.InvariantCulture))) + throw new InvalidDataException($"Invalid {_toolName} cache output inventory."); + + bool present = manifest.ReadBoolean(); + if (!present && !fileSet) + throw new InvalidDataException($"Directory inventory contains an absent {_toolName} cache file."); + if (present) + { + long length = manifest.ReadInt64(); + byte[] expectedHash = manifest.ReadBytes(32); + string cachedFile = Path.Combine(outputsDirectory, relativePath); + if (expectedHash.Length != 32 || new FileInfo(cachedFile).Length != length || + !expectedHash.SequenceEqual(HashFile(cachedFile))) + throw new InvalidDataException($"Invalid {_toolName} cache output: '{relativePath}'."); + } + files.Add((relativePath, present)); + } + + if (manifest.BaseStream.Position != manifest.BaseStream.Length) + throw new InvalidDataException($"Unexpected data in {_toolName} cache manifest."); + + return files; + } + + private void ValidateRelativePath(string path) + { + if (string.IsNullOrEmpty(path) || Path.IsPathRooted(path)) + throw new InvalidDataException($"Invalid {_toolName} cache output path."); + + foreach (string component in path.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar)) + { + if (component is "" or "." or ".." || component.IndexOfAny(Path.GetInvalidFileNameChars()) >= 0 || + (RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && component[component.Length - 1] is '.' or ' ')) + throw new InvalidDataException($"Invalid {_toolName} cache output path."); + } + } + + private static byte[] HashFile(string path) + { + using var sha256 = SHA256.Create(); + using var stream = File.OpenRead(path); + return sha256.ComputeHash(stream); + } + + private static bool IsCacheFailure(Exception ex) => ex is IOException or UnauthorizedAccessException or InvalidDataException or FormatException; +} diff --git a/src/tasks/Common/InvocationCache/InvocationCacheConfiguration.cs b/src/tasks/Common/InvocationCache/InvocationCacheConfiguration.cs new file mode 100644 index 00000000000000..f6e836c2368f12 --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationCacheConfiguration.cs @@ -0,0 +1,52 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.IO; +using System.Runtime.InteropServices; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed class InvocationCacheConfiguration +{ + internal string ToolName { get; } + internal string EnabledEnvironmentVariable { get; } + internal string PathEnvironmentVariable { get; } + + internal InvocationCacheConfiguration(string toolName, string enabledEnvironmentVariable, string pathEnvironmentVariable) + { + ToolName = toolName; + EnabledEnvironmentVariable = enabledEnvironmentVariable; + PathEnvironmentVariable = pathEnvironmentVariable; + } + + internal static string? GetDefaultCacheDirectory(string toolName) + { + string cacheDirectoryName = toolName.ToLowerInvariant(); + if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) + { + string localAppData = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData, Environment.SpecialFolderOption.DoNotVerify); + return string.IsNullOrEmpty(localAppData) || !Path.IsPathRooted(localAppData) + ? null + : Path.Combine(localAppData, cacheDirectoryName); + } + + bool isMacOS = RuntimeInformation.IsOSPlatform(OSPlatform.OSX); + if (!isMacOS) + { + string? cacheHome = Environment.GetEnvironmentVariable("XDG_CACHE_HOME"); + if (!string.IsNullOrEmpty(cacheHome) && Path.IsPathRooted(cacheHome)) + return Path.Combine(cacheHome, cacheDirectoryName); + } + + string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile, Environment.SpecialFolderOption.DoNotVerify); + if (string.IsNullOrEmpty(home) || !Path.IsPathRooted(home)) + return null; + + return isMacOS + ? Path.Combine(home, "Library", "Caches", cacheDirectoryName) + : Path.Combine(home, ".cache", cacheDirectoryName); + } +} diff --git a/src/tools/illink/src/ILLink.Tasks/ILLinkCacheEntry.cs b/src/tasks/Common/InvocationCache/InvocationCacheEntry.cs similarity index 70% rename from src/tools/illink/src/ILLink.Tasks/ILLinkCacheEntry.cs rename to src/tasks/Common/InvocationCache/InvocationCacheEntry.cs index 155ab5bf993d8c..d3a2924e94e1f9 100644 --- a/src/tools/illink/src/ILLink.Tasks/ILLinkCacheEntry.cs +++ b/src/tasks/Common/InvocationCache/InvocationCacheEntry.cs @@ -1,13 +1,14 @@ -// Copyright (c) .NET Foundation and contributors. All rights reserved. -// Licensed under the MIT license. See LICENSE file in the project root for full license information. +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. using System.Globalization; using System.IO; -namespace ILLink.Tasks; +namespace Microsoft.NET.Build.Tasks; -internal static class ILLinkCacheEntry +internal static class InvocationCacheEntry { + internal const int FormatVersion = 1; internal const string VersionDirectory = "v1"; internal const string LastUsedFileName = "last-used"; diff --git a/src/tools/illink/src/ILLink.CacheTool/CachePurge.cs b/src/tasks/Common/InvocationCache/InvocationCachePurge.cs similarity index 60% rename from src/tools/illink/src/ILLink.CacheTool/CachePurge.cs rename to src/tasks/Common/InvocationCache/InvocationCachePurge.cs index a7e4ebfc21b21b..f966969fe9783d 100644 --- a/src/tools/illink/src/ILLink.CacheTool/CachePurge.cs +++ b/src/tasks/Common/InvocationCache/InvocationCachePurge.cs @@ -1,31 +1,33 @@ -// Copyright (c) .NET Foundation and contributors. All rights reserved. -// Licensed under the MIT license. See LICENSE file in the project root for full license information. +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable using System; using System.Globalization; using System.IO; -using ILLink.Tasks; -namespace ILLink.CacheTool; +namespace Microsoft.NET.Build.Tasks; -internal static class CachePurge +internal static class InvocationCachePurge { - internal static int Run(string cacheDirectory, DateTimeOffset cutoff, TextWriter output, TextWriter error) + internal static int Run(string cacheDirectory, string versionDirectory, string toolName, + DateTimeOffset cutoff, TextWriter output, TextWriter error) { int deleted = 0; int kept = 0; int errors = 0; - string entriesDirectory = Path.Combine(cacheDirectory, ILLinkCacheEntry.VersionDirectory); + string entriesDirectory = Path.Combine(cacheDirectory, versionDirectory); try { foreach (string entry in Directory.EnumerateDirectories(entriesDirectory)) { - if (!ILLinkCacheEntry.IsEntryName(Path.GetFileName(entry))) + if (!InvocationCacheEntry.IsEntryName(Path.GetFileName(entry))) continue; try { - DateTimeOffset lastUsed = GetLastUsed(entry, error); + DateTimeOffset lastUsed = GetLastUsed(entry, toolName, error); if (lastUsed >= cutoff) { @@ -39,23 +41,23 @@ internal static int Run(string cacheDirectory, DateTimeOffset cutoff, TextWriter catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or FormatException or InvalidDataException) { errors++; - error.WriteLine($"Could not purge ILLink cache entry '{entry}': {ex.Message}"); + error.WriteLine($"Could not purge {toolName} cache entry '{entry}': {ex.Message}"); } } } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) { errors++; - error.WriteLine($"Could not enumerate ILLink cache '{entriesDirectory}': {ex.Message}"); + error.WriteLine($"Could not enumerate {toolName} cache '{entriesDirectory}': {ex.Message}"); } - output.WriteLine($"ILLink cache purge: Deleted: {deleted}, Kept: {kept}, Errors: {errors}"); + output.WriteLine($"{toolName} cache purge: Deleted: {deleted}, Kept: {kept}, Errors: {errors}"); return errors == 0 ? 0 : 1; } - private static DateTimeOffset GetLastUsed(string entry, TextWriter error) + private static DateTimeOffset GetLastUsed(string entry, string toolName, TextWriter error) { - string marker = Path.Combine(entry, ILLinkCacheEntry.LastUsedFileName); + string marker = Path.Combine(entry, InvocationCacheEntry.LastUsedFileName); string reason = "missing or invalid"; try { @@ -72,7 +74,7 @@ private static DateTimeOffset GetLastUsed(string entry, TextWriter error) reason = ex.Message; } - error.WriteLine($"Could not read a valid last-used marker for ILLink cache entry '{entry}' ({reason}); using directory creation time."); + error.WriteLine($"Could not read a valid last-used marker for {toolName} cache entry '{entry}' ({reason}); using directory creation time."); return new DateTimeOffset(Directory.GetCreationTimeUtc(entry), TimeSpan.Zero); } } diff --git a/src/tasks/Common/InvocationCache/InvocationDescription.cs b/src/tasks/Common/InvocationCache/InvocationDescription.cs new file mode 100644 index 00000000000000..4cae204df8dd99 --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationDescription.cs @@ -0,0 +1,20 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +namespace Microsoft.NET.Build.Tasks; + +internal sealed class InvocationDescription +{ + internal InvocationInputs Inputs { get; } + internal string[] Context { get; } + internal InvocationOutputs Outputs { get; } + + internal InvocationDescription(InvocationInputs inputs, string[] context, InvocationOutputs outputs) + { + Inputs = inputs; + Context = context; + Outputs = outputs; + } +} diff --git a/src/tasks/Common/InvocationCache/InvocationInputs.cs b/src/tasks/Common/InvocationCache/InvocationInputs.cs new file mode 100644 index 00000000000000..64c15afafb3b1d --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationInputs.cs @@ -0,0 +1,22 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.IO; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed class InvocationInputs +{ + internal string[] Files { get; } + + private InvocationInputs(string[] files) + { + Files = files; + } + + internal static InvocationInputs FromFiles(params string[] files) => + new InvocationInputs(Array.ConvertAll(files, Path.GetFullPath)); +} diff --git a/src/tasks/Common/InvocationCache/InvocationOutputs.cs b/src/tasks/Common/InvocationCache/InvocationOutputs.cs new file mode 100644 index 00000000000000..9437cf1b06475f --- /dev/null +++ b/src/tasks/Common/InvocationCache/InvocationOutputs.cs @@ -0,0 +1,27 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +#nullable enable + +using System; +using System.IO; + +namespace Microsoft.NET.Build.Tasks; + +internal sealed class InvocationOutputs +{ + internal string? DirectoryPath { get; } + internal string[] Files { get; } + + private InvocationOutputs(string? directoryPath, string[] files) + { + DirectoryPath = directoryPath; + Files = files; + } + + internal static InvocationOutputs FromDirectory(string path) => + new InvocationOutputs(Path.GetFullPath(path), Array.Empty()); + + internal static InvocationOutputs FromFiles(params string[] files) => + new InvocationOutputs(null, Array.ConvertAll(files, Path.GetFullPath)); +} diff --git a/src/tasks/Common/InvocationCache/README.md b/src/tasks/Common/InvocationCache/README.md new file mode 100644 index 00000000000000..086685848a9953 --- /dev/null +++ b/src/tasks/Common/InvocationCache/README.md @@ -0,0 +1,47 @@ +# Experimental invocation cache + +These sources cache successful MSBuild task invocations and are compiled into each consumer. The API +and storage format are experimental, without compatibility guarantees. + +## Integrating a task + +Call `InvocationCache.ExecuteCached` with: + +- An `InvocationCacheConfiguration` naming the tool and its enable/path environment variables. Only + `true` enables caching; a path alone does not. Without an override, storage uses the platform's user + cache directory. +- A description callback returning ordered input paths, effective arguments/context, and an owned + directory or file set through `InvocationDescription`. Return null for unsupported invocations. +- A normal-execution callback that prepares outputs and runs the tool independently of caching. + +The task must inventory all relevant inputs and toolchain files, including its own assembly. The +shared layer hashes declared files with SHA-256; it does not interpret arguments or discover +dependencies. Inputs must remain stable through discovery, hashing, and execution. Inherited +environment settings and installed runtime trees are not automatically fingerprinted. + +## Behavior and limitations + +Disabled caching skips discovery and cache I/O. Hits validate and restore outputs without running the +tool. Misses and recoverable cache failures run it normally; only exit code zero with no logged errors +is stored. Failures are diagnosed, and failed tool execution is never retried. Invalid output +declarations, including cache/output overlap, fail before cleanup rather than bypassing caching. + +Normal execution owns output preparation, including removal of stale or partially restored files. +The cache owns reconciliation only on hits, including deleting declared files recorded as absent. +Restoration is not transactional. Empty directories are not preserved, and symbolic links are not +checked. + +Keys retain absolute input/output paths and execution context; cross-worktree reuse is not promised. +Hits do not replay diagnostics or non-file task outputs. Cache operations do not observe cancellation. +Consumers must bypass invocations whose dependencies or required side effects cannot be modeled. + +Entries are staged locally and published atomically. Concurrent processes on one machine may share +storage when writing distinct destinations; same-destination execution and distributed writers are +unsupported. + +There is no automatic eviction. Corrupt or incompatible entries are not repaired; obsolete entries +within the current `v1` layout remain purgable. Purge or manual deletion must not race active users. + +See the [ILLink integration](../../../../docs/design/tools/illink/task-cache.md) for eligibility and +toolchain omissions, and the [maintenance tool README](../../../tools/BuildCache/README.md) for purge +behavior and commands. diff --git a/src/tools/BuildCache/README.md b/src/tools/BuildCache/README.md new file mode 100644 index 00000000000000..5c0570486152b1 --- /dev/null +++ b/src/tools/BuildCache/README.md @@ -0,0 +1,53 @@ +# Experimental build invocation cache maintenance + +`dotnet-build-cache` is a .NET tool for maintaining experimental MSBuild invocation caches. +Its commands and the cache format may change without notice. Use the tool built from the same +revision as the task. It does not manage Roslyn/csc or native compiler caches. +The [shared cache README](../../tasks/Common/InvocationCache/README.md) describes the task contract, +storage format, and lifecycle; this document covers the maintenance CLI. + +```sh +dotnet build-cache purge --cache illink --before 2026-10-01T12:00:00Z +dotnet build-cache purge --cache crossgen2 --before 2026-10-01T12:00:00Z +dotnet build-cache purge --cache-directory /path/to/cache --before 2026-10-01T12:00:00Z +``` + +Choose `--cache` for a named cache or `--cache-directory` to select a directory directly. +Named caches use the corresponding `ILLINK_EXPERIMENTAL_CACHE_PATH` or +`CROSSGEN2_EXPERIMENTAL_CACHE_PATH`, falling back to the task's platform default (`illink` +or `crossgen2` under the user cache directory) when the environment variable is unset or empty. +If both options are supplied, `--cache-directory` determines the target; the name is validated +and only affects diagnostic labels. It does not check that the directory belongs to that cache. +An unknown name or invalid selected directory is an error; the tool does not fall back from +an invalid path or create missing directories. Maintenance ignores the boolean cache-enablement +environment variables, so disabled caches can still be maintained. + +`--before` is required and must be an ISO 8601 UTC timestamp, with `Z` or `+00:00`. +Entries last used strictly before the cutoff are deleted; entries at or after it are kept. +The tool prints deleted, kept, and error counts. Argument or maintenance errors return a +nonzero exit code. Missing, invalid, or unreadable usage markers use the entry directory's +creation time instead; this fallback is reported on stderr but does not count as an error. + +**No build may use the cache during purging.** The tool does not coordinate with active +readers or writers. Staging directories and unrelated directories are left alone. +There is no migration of entries created before usage markers were introduced. +The tool maintains only the shared `v1` directory. It does not read manifests, so +obsolete or incompatible entries within `v1` can be purged using the same cutoff. +Other version directories are left untouched. + +## Building locally + +From the repository root: + +```sh +./build.sh tools.buildcache -c Release +./dotnet.sh pack src/tools/BuildCache/src/BuildCache.csproj -c Release +``` + +The package is produced under `artifacts/packages/Release/NonShipping`. Install the exact +locally built version with `dotnet tool install --tool-path +--add-source artifacts/packages/Release/NonShipping dotnet-build-cache --version `, +then add `` to `PATH` or invoke its `dotnet-build-cache` executable directly. +Alternatively, run the built `artifacts/bin/BuildCache/Release/net*/dotnet-build-cache.dll` +with `dotnet` directly. The `tools.buildcachetests` subset builds its tests; the existing +`tools.illink` and `tools.illinktests` subsets also include the tool and tests respectively. diff --git a/src/tools/BuildCache/src/BuildCache.csproj b/src/tools/BuildCache/src/BuildCache.csproj new file mode 100644 index 00000000000000..4faee06d959997 --- /dev/null +++ b/src/tools/BuildCache/src/BuildCache.csproj @@ -0,0 +1,26 @@ + + + $(NetCoreAppToolCurrent) + enable + dotnet-build-cache + Exe + false + Major + true + false + true + dotnet-build-cache + dotnet-build-cache + Experimental build invocation cache maintenance tool. + README.md + + + + + + + + + + + diff --git a/src/tools/BuildCache/src/Program.cs b/src/tools/BuildCache/src/Program.cs new file mode 100644 index 00000000000000..e51deac2255e8d --- /dev/null +++ b/src/tools/BuildCache/src/Program.cs @@ -0,0 +1,111 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.CommandLine; +using System.Globalization; +using System.IO; +using Microsoft.NET.Build.Tasks; + +namespace BuildCache; + +internal static class Program +{ + private static readonly string[] s_cutoffFormats = { "yyyy-MM-dd'T'HH:mm:ss.FFFFFFF'Z'", "yyyy-MM-dd'T'HH:mm:ss.FFFFFFFzzz" }; + + private static int Main(string[] args) => Run(args, Console.Out, Console.Error); + + internal static int Run(string[] args, TextWriter output, TextWriter error) + { + var command = new RootCommand("Experimental build invocation cache maintenance. Commands and cache formats may change without notice."); + var cache = new Option("--cache") + { + Description = "Named cache: illink or crossgen2. Uses its path environment variable or platform default." + }; + var directory = new Option("--cache-directory") + { + Description = "Explicit cache root, overriding named-cache settings. No builds may use this cache while purging." + }.AcceptExistingOnly(); + var before = new Option("--before") + { + Description = "Delete entries last used before this UTC ISO 8601 timestamp (Z or +00:00).", + Required = true, + CustomParser = result => + { + string value = result.Tokens[0].Value; + if (DateTimeOffset.TryParseExact(value, s_cutoffFormats, + CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out DateTimeOffset cutoff) && + cutoff.Offset == TimeSpan.Zero) + { + return cutoff; + } + + result.AddError("--before must be a UTC ISO 8601 timestamp ending in Z or +00:00."); + return default; + } + }; + var purge = new Command("purge", "Delete unused entries from an idle invocation cache."); + purge.Options.Add(cache); + purge.Options.Add(directory); + purge.Options.Add(before); + purge.SetAction(result => + { + string? name = result.GetValue(cache); + InvocationCacheConfiguration? configuration = name switch + { + "illink" => new("ILLink", "ILLINK_EXPERIMENTAL_CACHE", "ILLINK_EXPERIMENTAL_CACHE_PATH"), + "crossgen2" => new("Crossgen2", "CROSSGEN2_EXPERIMENTAL_CACHE", "CROSSGEN2_EXPERIMENTAL_CACHE_PATH"), + _ => null + }; + if (name is not null && configuration is null) + { + error.WriteLine("--cache must be 'illink' or 'crossgen2'."); + return 1; + } + + string? path = result.GetValue(directory)?.FullName; + if (path is null && configuration is null) + { + error.WriteLine("Specify --cache or --cache-directory."); + return 1; + } + + try + { + if (path is null && configuration is not null) + { + path = Environment.GetEnvironmentVariable(configuration.PathEnvironmentVariable); + if (string.IsNullOrEmpty(path)) + path = InvocationCacheConfiguration.GetDefaultCacheDirectory(configuration.ToolName); + } + if (path is null) + { + error.WriteLine("No default cache directory is available; specify --cache-directory."); + return 1; + } + + path = Path.GetFullPath(path); + if (!Directory.Exists(path)) + { + error.WriteLine($"Cache directory '{path}' does not exist or cannot be accessed."); + return 1; + } + } + catch (Exception ex) when (ex is ArgumentException or NotSupportedException or IOException or UnauthorizedAccessException) + { + error.WriteLine($"Could not resolve cache directory '{path}': {ex.Message}"); + return 1; + } + + return InvocationCachePurge.Run(path, InvocationCacheEntry.VersionDirectory, + configuration?.ToolName ?? "Build", result.GetValue(before), output, error); + }); + command.Subcommands.Add(purge); + return command.Parse(args).Invoke(new() + { + EnableDefaultExceptionHandler = false, + Output = output, + Error = error + }); + } +} diff --git a/src/tools/BuildCache/tests/BuildCache.Tests.csproj b/src/tools/BuildCache/tests/BuildCache.Tests.csproj new file mode 100644 index 00000000000000..d25102424d7538 --- /dev/null +++ b/src/tools/BuildCache/tests/BuildCache.Tests.csproj @@ -0,0 +1,11 @@ + + + $(NetCoreAppToolCurrent) + + + + + + + + diff --git a/src/tools/BuildCache/tests/CachePurgeTests.cs b/src/tools/BuildCache/tests/CachePurgeTests.cs new file mode 100644 index 00000000000000..c94d07c9142421 --- /dev/null +++ b/src/tools/BuildCache/tests/CachePurgeTests.cs @@ -0,0 +1,283 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System; +using System.IO; +using System.Security.Cryptography; +using Microsoft.DotNet.RemoteExecutor; +using Microsoft.NET.Build.Tasks; +using Xunit; +using TestCache = Microsoft.NET.Build.Tasks.Tests.InvocationCachePurgeTests.TestCache; + +namespace BuildCache.Tests; + +public class CachePurgeTests +{ + [Theory] + [InlineData("2025-01-02T03:04:05Z")] + [InlineData("2025-01-02T03:04:05+00:00")] + [InlineData("2025-01-02T03:04:05.0000000+00:00")] + [InlineData("2025-01-02T03:04:05.0000000Z")] + public void CommandAcceptsExplicitUtcCutoff(string cutoff) + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddTicks(-1)); + + Assert.Equal(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", cutoff }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(entry)); + Assert.Equal($"Build cache purge: Deleted: 1, Kept: 0, Errors: 0{Environment.NewLine}", cache.Output.ToString()); + Assert.Empty(cache.Error.ToString()); + } + + [Theory] + [InlineData("v2")] + [InlineData("other-layout")] + public void CommandOnlyPurgesCurrentLayout(string otherVersion) + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddTicks(-1)); + string otherEntry = Path.Combine(cache.Root, otherVersion, new string('b', 64)); + Directory.CreateDirectory(otherEntry); + File.WriteAllText(Path.Combine(otherEntry, InvocationCacheEntry.LastUsedFileName), + "2020-01-01T00:00:00.0000000+00:00"); + + Assert.Equal(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(entry)); + Assert.True(Directory.Exists(otherEntry)); + Assert.Equal($"Build cache purge: Deleted: 1, Kept: 0, Errors: 0{Environment.NewLine}", cache.Output.ToString()); + Assert.Empty(cache.Error.ToString()); + } + + [Fact] + public void CommandPurgesLegacyV1Entries() + { + using var cache = new TestCache("v1"); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddTicks(-1)); + byte[] image = File.ReadAllBytes(Path.Combine(entry, "outputs", "app.dll")); + using (var manifest = new BinaryWriter(File.Create(Path.Combine(entry, "manifest")))) + { + manifest.Write(1); + manifest.Write(1); + manifest.Write("app.dll"); + manifest.Write((long)image.Length); + manifest.Write(SHA256.HashData(image)); + } + Assert.False(File.Exists(Path.Combine(entry, "manifest.sha256"))); + + Assert.Equal(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(entry)); + Assert.Equal($"Build cache purge: Deleted: 1, Kept: 0, Errors: 0{Environment.NewLine}", cache.Output.ToString()); + Assert.Empty(cache.Error.ToString()); + } + + [Theory] + [InlineData("invalid")] + [InlineData("")] + [InlineData("2025-01-02")] + [InlineData("2025-01-02T03:04:05")] + [InlineData("2025-01-02T03:04:05+01:00")] + public void CommandRejectsInvalidCutoffWithoutDeleting(string cutoff) + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + + Assert.NotEqual(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", cutoff }, cache.Output, cache.Error)); + + Assert.True(Directory.Exists(entry)); + Assert.Contains("--before", cache.Error.ToString()); + Assert.DoesNotContain("Build cache purge:", cache.Output.ToString()); + } + + [Theory] + [InlineData("missing-directory")] + [InlineData("missing-cutoff")] + [InlineData("missing-value")] + [InlineData("unknown-option")] + [InlineData("unknown-command")] + public void CommandRejectsIncompleteOrUnknownArguments(string scenario) + { + using var cache = new TestCache(); + string[] args = scenario switch + { + "missing-directory" => new[] { "purge", "--before", "2025-01-02T03:04:05Z" }, + "missing-cutoff" => new[] { "purge", "--cache-directory", cache.Root }, + "missing-value" => new[] { "purge", "--cache-directory", cache.Root, "--before" }, + "unknown-option" => new[] { "purge", "--cache-directory", cache.Root, "--before", "2025-01-02T03:04:05Z", "--force" }, + _ => new[] { "delete-all" } + }; + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + + Assert.NotEqual(0, Program.Run(args, cache.Output, cache.Error)); + + Assert.True(Directory.Exists(entry)); + Assert.NotEmpty(cache.Error.ToString()); + } + + [Fact] + public void CommandRejectsNonexistentDirectory() + { + using var cache = new TestCache(); + string missing = Path.Combine(cache.Root, "missing"); + + Assert.NotEqual(0, Program.Run(new[] { "purge", "--cache-directory", missing, "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.NotEmpty(cache.Error.ToString()); + Assert.False(Directory.Exists(missing)); + } + + [Fact] + public void CommandPropagatesMaintenanceFailure() + { + using var cache = new TestCache(); + Directory.Delete(cache.Entries); + + Assert.Equal(1, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.Contains("Errors: 1", cache.Output.ToString()); + Assert.StartsWith($"Could not enumerate Build cache '{cache.Entries}':", cache.Error.ToString()); + } + + [Theory] + [InlineData("unknown", false)] + [InlineData("unknown", true)] + [InlineData("", true)] + [InlineData("ILLink", true)] + public void CommandRejectsUnknownCacheEvenWithDirectory(string name, bool explicitDirectory) + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + string[] args = explicitDirectory + ? new[] { "purge", "--cache", name, "--cache-directory", cache.Root, "--before", "2025-01-02T03:04:05Z" } + : new[] { "purge", "--cache", name, "--before", "2025-01-02T03:04:05Z" }; + + Assert.Equal(1, Program.Run(args, cache.Output, cache.Error)); + + Assert.True(Directory.Exists(entry)); + Assert.Contains("--cache must be 'illink' or 'crossgen2'.", cache.Error.ToString()); + Assert.Empty(cache.Output.ToString()); + } + + [Theory] + [InlineData("illink", "true")] + [InlineData("illink", "false")] + [InlineData("illink", "invalid")] + [InlineData("illink", "")] + [InlineData("crossgen2", "true")] + [InlineData("crossgen2", "false")] + [InlineData("crossgen2", "invalid")] + [InlineData("crossgen2", "")] + public void NamedCacheUsesPathEnvironmentRegardlessOfEnablement(string name, string enabled) => + RemoteExecutor.Invoke(static (cacheName, enabledValue) => + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + string prefix = cacheName.ToUpperInvariant() + "_EXPERIMENTAL_CACHE"; + Environment.SetEnvironmentVariable(prefix, enabledValue == "" ? null : enabledValue); + Environment.SetEnvironmentVariable(prefix + "_PATH", cache.Root); + Environment.SetEnvironmentVariable("XDG_CACHE_HOME", Path.Combine(cache.Root, "unused-default")); + + Assert.Equal(0, Program.Run(new[] { "purge", "--cache", cacheName, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(entry)); + string label = cacheName == "illink" ? "ILLink" : "Crossgen2"; + Assert.Equal($"{label} cache purge: Deleted: 1, Kept: 0, Errors: 0{Environment.NewLine}", cache.Output.ToString()); + Assert.Empty(cache.Error.ToString()); + Assert.False(Directory.Exists(Path.Combine(cache.Root, "unused-default"))); + }, name, enabled).Dispose(); + + [Theory] + [InlineData("illink", false)] + [InlineData("illink", true)] + [InlineData("crossgen2", false)] + [InlineData("crossgen2", true)] + public void ExplicitDirectoryOverridesNamedEnvironmentWithoutFallback(string name, bool missing) => + RemoteExecutor.Invoke(static (cacheName, missingValue) => + { + using var cache = new TestCache(); + using var environmentCache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + string environmentEntry = environmentCache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + Environment.SetEnvironmentVariable(cacheName.ToUpperInvariant() + "_EXPERIMENTAL_CACHE_PATH", environmentCache.Root); + bool missingDirectory = bool.Parse(missingValue); + string directory = missingDirectory ? Path.Combine(cache.Root, "missing") : cache.Root; + + int exitCode = Program.Run(new[] { "purge", "--cache", cacheName, "--cache-directory", directory, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error); + + Assert.Equal(missingDirectory, exitCode != 0); + Assert.Equal(missingDirectory, Directory.Exists(entry)); + Assert.True(Directory.Exists(environmentEntry)); + Assert.Equal(!missingDirectory, Directory.Exists(directory)); + if (missingDirectory) + Assert.NotEmpty(cache.Error.ToString()); + else + Assert.Empty(cache.Error.ToString()); + }, name, missing.ToString()).Dispose(); + + [Theory] + [InlineData("illink", false)] + [InlineData("illink", true)] + [InlineData("crossgen2", false)] + [InlineData("crossgen2", true)] + public void NamedCacheRejectsMissingOrFileDirectoryWithoutCreatingIt(string name, bool file) => + RemoteExecutor.Invoke(static (cacheName, fileValue) => + { + using var cache = new TestCache(); + string invalid = Path.Combine(cache.Root, "invalid"); + bool isFile = bool.Parse(fileValue); + if (isFile) + File.WriteAllText(invalid, "not a directory"); + Environment.SetEnvironmentVariable(cacheName.ToUpperInvariant() + "_EXPERIMENTAL_CACHE_PATH", invalid); + + Assert.Equal(1, Program.Run(new[] { "purge", "--cache", cacheName, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(invalid)); + Assert.Equal(isFile, File.Exists(invalid)); + Assert.Contains(invalid, cache.Error.ToString()); + Assert.Empty(cache.Output.ToString()); + }, name, file.ToString()).Dispose(); + + [ConditionalTheory(typeof(OperatingSystem), nameof(OperatingSystem.IsLinux))] + [PlatformSpecific(TestPlatforms.Linux)] + [InlineData("illink", "")] + [InlineData("illink", "")] + [InlineData("crossgen2", "")] + [InlineData("crossgen2", "")] + public void NamedCacheUsesPlatformDefaultWhenPathIsUnsetOrEmpty(string name, string path) => + RemoteExecutor.Invoke(static (cacheName, pathValue) => + { + using var cache = new TestCache(); + string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); + string root = Path.Combine(cache.Root, cacheName); + Directory.CreateDirectory(root); + Directory.Move(cache.Entries, Path.Combine(root, InvocationCacheEntry.VersionDirectory)); + entry = Path.Combine(root, InvocationCacheEntry.VersionDirectory, Path.GetFileName(entry)); + Environment.SetEnvironmentVariable("XDG_CACHE_HOME", cache.Root); + Environment.SetEnvironmentVariable(cacheName.ToUpperInvariant() + "_EXPERIMENTAL_CACHE_PATH", + pathValue == "" ? null : pathValue); + + Assert.Equal(root, InvocationCacheConfiguration.GetDefaultCacheDirectory(cacheName)); + Assert.Equal(0, Program.Run(new[] { "purge", "--cache", cacheName, + "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); + + Assert.False(Directory.Exists(entry)); + Assert.Empty(cache.Error.ToString()); + }, name, path).Dispose(); + + [Fact] + public void HelpExplainsIdleCacheRequirement() + { + using var output = new StringWriter(); + using var error = new StringWriter(); + Assert.Equal(0, Program.Run(new[] { "purge", "--help" }, output, error)); + Assert.Contains("No builds may use this cache while purging.", output.ToString()); + Assert.Empty(error.ToString()); + } +} diff --git a/src/tools/illink/illink.slnx b/src/tools/illink/illink.slnx index bda8cde96cef07..ea2615feacf506 100644 --- a/src/tools/illink/illink.slnx +++ b/src/tools/illink/illink.slnx @@ -13,7 +13,7 @@ - + @@ -21,7 +21,7 @@ - + diff --git a/src/tools/illink/src/ILLink.CacheTool/ILLink.CacheTool.csproj b/src/tools/illink/src/ILLink.CacheTool/ILLink.CacheTool.csproj deleted file mode 100644 index 6a6d45539fb368..00000000000000 --- a/src/tools/illink/src/ILLink.CacheTool/ILLink.CacheTool.csproj +++ /dev/null @@ -1,22 +0,0 @@ - - - dotnet-illink-cache - Exe - false - Major - true - false - true - dotnet-illink-cache - dotnet-illink-cache - Experimental ILLink task cache maintenance tool. - README.md - - - - - - - - - diff --git a/src/tools/illink/src/ILLink.CacheTool/Program.cs b/src/tools/illink/src/ILLink.CacheTool/Program.cs deleted file mode 100644 index 6cf7a19cb729d4..00000000000000 --- a/src/tools/illink/src/ILLink.CacheTool/Program.cs +++ /dev/null @@ -1,55 +0,0 @@ -// Copyright (c) .NET Foundation and contributors. All rights reserved. -// Licensed under the MIT license. See LICENSE file in the project root for full license information. - -using System; -using System.CommandLine; -using System.Globalization; -using System.IO; - -namespace ILLink.CacheTool; - -internal static class Program -{ - private static readonly string[] s_cutoffFormats = { "yyyy-MM-dd'T'HH:mm:ss.FFFFFFF'Z'", "yyyy-MM-dd'T'HH:mm:ss.FFFFFFFzzz" }; - - private static int Main(string[] args) => Run(args, Console.Out, Console.Error); - - internal static int Run(string[] args, TextWriter output, TextWriter error) - { - var command = new RootCommand("Experimental ILLink task cache maintenance. Commands and cache formats may change without notice."); - var directory = new Option("--cache-directory") - { - Description = "Cache root directory. No builds may use this cache while purging.", - Required = true - }.AcceptExistingOnly(); - var before = new Option("--before") - { - Description = "Delete entries last used before this UTC ISO 8601 timestamp (Z or +00:00).", - Required = true, - CustomParser = result => - { - string value = result.Tokens[0].Value; - if (DateTimeOffset.TryParseExact(value, s_cutoffFormats, - CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out DateTimeOffset cutoff) && - cutoff.Offset == TimeSpan.Zero) - { - return cutoff; - } - - result.AddError("--before must be a UTC ISO 8601 timestamp ending in Z or +00:00."); - return default; - } - }; - var purge = new Command("purge", "Delete unused entries from an idle ILLink task cache."); - purge.Options.Add(directory); - purge.Options.Add(before); - purge.SetAction(result => CachePurge.Run(result.GetValue(directory)!.FullName, result.GetValue(before), output, error)); - command.Subcommands.Add(purge); - return command.Parse(args).Invoke(new() - { - EnableDefaultExceptionHandler = false, - Output = output, - Error = error - }); - } -} diff --git a/src/tools/illink/src/ILLink.CacheTool/README.md b/src/tools/illink/src/ILLink.CacheTool/README.md deleted file mode 100644 index e8dfd8b679f94d..00000000000000 --- a/src/tools/illink/src/ILLink.CacheTool/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Experimental ILLink cache maintenance - -`dotnet-illink-cache` is a .NET tool for maintaining the experimental ILLink MSBuild task cache. -Its commands and the cache format may change without notice. Use the tool built from the same -revision as the task. It does not manage Roslyn/csc or native compiler caches. - -```sh -dotnet illink-cache purge --cache-directory /path/to/cache --before 2026-10-01T12:00:00Z -``` - -Both options are required. The cutoff must be an ISO 8601 UTC timestamp, with `Z` or `+00:00`. -Entries last used strictly before the cutoff are deleted; entries at or after it are kept. -The tool prints deleted, kept, and error counts. Argument or maintenance errors return a -nonzero exit code. Missing, invalid, or unreadable usage markers use the entry directory's -creation time instead; this fallback is reported on stderr but does not count as an error. - -**No build may use the cache during purging.** The tool does not coordinate with active -readers or writers. Staging directories and unrelated directories are left alone. -There is no migration of entries created before usage markers were introduced. - -## Building locally - -From the repository root: - -```sh -./dotnet.sh pack src/tools/illink/src/ILLink.CacheTool/ILLink.CacheTool.csproj -c Release -``` - -The package is produced under `artifacts/packages/Release/NonShipping`. Install the exact -locally built version with `dotnet tool install --tool-path ---add-source artifacts/packages/Release/NonShipping dotnet-illink-cache --version `, -then add `` to `PATH` or invoke its `dotnet-illink-cache` executable directly. -Alternatively, run the built `dotnet-illink-cache.dll` with `dotnet` directly. diff --git a/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj b/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj index 6117c8feac1e67..141cc75a048f1a 100644 --- a/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj +++ b/src/tools/illink/src/ILLink.Tasks/ILLink.Tasks.csproj @@ -50,6 +50,15 @@ + + + + + + + + + diff --git a/src/tools/illink/src/ILLink.Tasks/ILLinkCache.cs b/src/tools/illink/src/ILLink.Tasks/ILLinkCache.cs deleted file mode 100644 index 6932c6a6bfc0e2..00000000000000 --- a/src/tools/illink/src/ILLink.Tasks/ILLinkCache.cs +++ /dev/null @@ -1,346 +0,0 @@ -// Copyright (c) .NET Foundation and contributors. All rights reserved. -// Licensed under the MIT license. See LICENSE file in the project root for full license information. - -#nullable enable - -using System; -using System.Collections.Generic; -using System.IO; -using System.Linq; -using System.Runtime.InteropServices; -using System.Security.Cryptography; -using System.Text; -using System.Threading; -using Microsoft.Build.Framework; -using Microsoft.Build.Utilities; - -namespace ILLink.Tasks; - -internal sealed class ILLinkCache -{ - private const int FormatVersion = 1; - private readonly TaskLoggingHelper _log; - private static readonly StringComparison s_pathComparison = RuntimeInformation.IsOSPlatform(OSPlatform.Windows) - ? StringComparison.OrdinalIgnoreCase - : StringComparison.Ordinal; - - internal string CacheDirectory { get; } - - private ILLinkCache(string cacheDirectory, TaskLoggingHelper log) - { - CacheDirectory = cacheDirectory; - _log = log; - } - - internal static ILLinkCache? TryCreateFromEnvironment(TaskLoggingHelper log) - { - string? setting = Environment.GetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE"); - if (string.IsNullOrEmpty(setting)) - return null; - - if (!bool.TryParse(setting, out bool enabled)) - { - log.LogMessage(MessageImportance.Low, "ILLink caching is disabled because ILLINK_EXPERIMENTAL_CACHE must be 'true' or 'false'."); - return null; - } - - return enabled ? TryCreate(Environment.GetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE_PATH"), log) : null; - } - - internal static ILLinkCache? TryCreate(string? cacheDirectory, TaskLoggingHelper log) - { - if (string.IsNullOrEmpty(cacheDirectory)) - cacheDirectory = GetDefaultCacheDirectory(); - - if (cacheDirectory is null) - { - log.LogMessage(MessageImportance.Low, "ILLink caching is disabled because no default cache directory is available."); - return null; - } - - try - { - return new ILLinkCache(Path.GetFullPath(cacheDirectory), log); - } - catch (Exception ex) when (ex is ArgumentException or NotSupportedException or IOException or UnauthorizedAccessException) - { - log.LogMessage(MessageImportance.Low, $"ILLink caching is disabled because the cache directory '{cacheDirectory}' could not be resolved: {ex.Message}"); - return null; - } - } - - private static string? GetDefaultCacheDirectory() - { - if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) - { - string localAppData = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData, Environment.SpecialFolderOption.DoNotVerify); - return string.IsNullOrEmpty(localAppData) || !Path.IsPathRooted(localAppData) - ? null - : Path.Combine(localAppData, "illink"); - } - - bool isMacOS = RuntimeInformation.IsOSPlatform(OSPlatform.OSX); - if (!isMacOS) - { - string? cacheHome = Environment.GetEnvironmentVariable("XDG_CACHE_HOME"); - if (!string.IsNullOrEmpty(cacheHome) && Path.IsPathRooted(cacheHome)) - return Path.Combine(cacheHome, "illink"); - } - - string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile, Environment.SpecialFolderOption.DoNotVerify); - if (string.IsNullOrEmpty(home) || !Path.IsPathRooted(home)) - return null; - - return isMacOS - ? Path.Combine(home, "Library", "Caches", "illink") - : Path.Combine(home, ".cache", "illink"); - } - - internal bool TryRestore(string inputHash, string outputDirectory, DateTime outputTimestampUtc) - { - string entryDirectory = GetEntryDirectory(inputHash); - outputDirectory = ValidateOutputDirectory(outputDirectory); - if (outputTimestampUtc.Kind != DateTimeKind.Utc) - throw new ArgumentException("The output timestamp must be UTC.", nameof(outputTimestampUtc)); - - if (!Directory.Exists(entryDirectory)) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache miss: {inputHash}"); - return false; - } - - try - { - List files = ReadManifest(entryDirectory); - - // Check the entire entry before modifying outputs. Copies can still fail if the cache is wiped concurrently. - if (Directory.Exists(outputDirectory)) - Directory.Delete(outputDirectory, recursive: true); - - Directory.CreateDirectory(outputDirectory); - foreach (string relativePath in files) - { - string destination = Path.Combine(outputDirectory, relativePath); - Directory.CreateDirectory(Path.GetDirectoryName(destination)!); - File.Copy(Path.Combine(entryDirectory, "outputs", relativePath), destination, overwrite: true); - File.SetLastWriteTimeUtc(destination, outputTimestampUtc); - } - - UpdateLastUsed(entryDirectory); - _log.LogMessage(MessageImportance.Low, $"ILLink cache hit: {inputHash}"); - return true; - } - catch (Exception ex) when (IsCacheFailure(ex)) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache restore failed for '{inputHash}'; running ILLink normally: {ex.Message}"); - return false; - } - } - - internal void Store(string inputHash, string outputDirectory) - { - string entryDirectory = GetEntryDirectory(inputHash); - outputDirectory = ValidateOutputDirectory(outputDirectory); - string? stagingDirectory = null; - - try - { - using var mutex = new Mutex(initiallyOwned: false, GetWriterMutexName(inputHash)); - bool acquired; - try - { - acquired = mutex.WaitOne(0); - } - catch (AbandonedMutexException) - { - acquired = true; - _log.LogMessage(MessageImportance.Low, $"ILLink cache writer lock was abandoned: {inputHash}"); - } - - if (!acquired) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache store skipped; another writer owns '{inputHash}'."); - return; - } - - try - { - if (Directory.Exists(entryDirectory)) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache store skipped; entry '{inputHash}' already exists."); - return; - } - - var files = new List(); - CollectOutputFiles(outputDirectory, "", files); - files.Sort(StringComparer.Ordinal); - - string parentDirectory = Path.GetDirectoryName(entryDirectory)!; - Directory.CreateDirectory(parentDirectory); - stagingDirectory = Path.Combine(parentDirectory, inputHash + "." + Guid.NewGuid().ToString("N") + ".tmp"); - Directory.CreateDirectory(stagingDirectory); - - using (var manifest = new BinaryWriter(File.Create(Path.Combine(stagingDirectory, "manifest")))) - { - manifest.Write(FormatVersion); - manifest.Write(files.Count); - foreach (string relativePath in files) - { - string cachedFile = Path.Combine(stagingDirectory, "outputs", relativePath); - Directory.CreateDirectory(Path.GetDirectoryName(cachedFile)!); - File.Copy(Path.Combine(outputDirectory, relativePath), cachedFile); - manifest.Write(relativePath); - manifest.Write(new FileInfo(cachedFile).Length); - manifest.Write(HashFile(cachedFile)); - } - } - - ILLinkCacheEntry.WriteLastUsed(Path.Combine(stagingDirectory, ILLinkCacheEntry.LastUsedFileName)); - Directory.Move(stagingDirectory, entryDirectory); - stagingDirectory = null; - _log.LogMessage(MessageImportance.Low, $"ILLink cache stored: {inputHash}"); - } - finally - { - mutex.ReleaseMutex(); - } - } - catch (Exception ex) when (IsCacheFailure(ex)) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache store failed for '{inputHash}': {ex.Message}"); - } - finally - { - if (stagingDirectory is not null) - { - try - { - if (Directory.Exists(stagingDirectory)) - Directory.Delete(stagingDirectory, recursive: true); - } - catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache staging cleanup failed for '{stagingDirectory}': {ex.Message}"); - } - } - } - } - - private void UpdateLastUsed(string entryDirectory) - { - try - { - ILLinkCacheEntry.WriteLastUsed(Path.Combine(entryDirectory, ILLinkCacheEntry.LastUsedFileName)); - } - catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) - { - _log.LogMessage(MessageImportance.Low, $"ILLink cache last-used update failed for '{entryDirectory}': {ex.Message}"); - } - } - - private string GetEntryDirectory(string inputHash) - { - if (!ILLinkCacheEntry.IsEntryName(inputHash)) - throw new ArgumentException("The input hash must be a lowercase SHA-256 hex string.", nameof(inputHash)); - - return Path.Combine(CacheDirectory, ILLinkCacheEntry.VersionDirectory, inputHash); - } - - private string GetWriterMutexName(string inputHash) - { - string directory = CacheDirectory.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar); - if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows)) - directory = directory.ToUpperInvariant(); - -#if NET - byte[] hash = SHA256.HashData(Encoding.UTF8.GetBytes(directory + "\n" + inputHash)); - return @"Global\ILLinkCache-" + Convert.ToHexString(hash); -#else - using var sha256 = SHA256.Create(); - byte[] hash = sha256.ComputeHash(Encoding.UTF8.GetBytes(directory + "\n" + inputHash)); - return @"Global\ILLinkCache-" + BitConverter.ToString(hash).Replace("-", ""); -#endif - } - - private string ValidateOutputDirectory(string outputDirectory) - { - string outputPath = Path.GetFullPath(outputDirectory); - string cachePrefix = CacheDirectory.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; - string outputPrefix = outputPath.TrimEnd(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar) + Path.DirectorySeparatorChar; - if (cachePrefix.StartsWith(outputPrefix, s_pathComparison) || outputPrefix.StartsWith(cachePrefix, s_pathComparison)) - throw new ArgumentException("The cache and output directories must not contain each other.", nameof(outputDirectory)); - - return outputPath; - } - - private static void CollectOutputFiles(string directory, string relativeDirectory, List files) - { - foreach (string path in Directory.EnumerateFileSystemEntries(directory)) - { - string relativePath = Path.Combine(relativeDirectory, Path.GetFileName(path)); - if (Directory.Exists(path)) - CollectOutputFiles(path, relativePath, files); - else - files.Add(relativePath); - } - } - - private static List ReadManifest(string entryDirectory) - { - using var manifest = new BinaryReader(File.OpenRead(Path.Combine(entryDirectory, "manifest"))); - if (manifest.ReadInt32() != FormatVersion) - throw new InvalidDataException("Unsupported ILLink cache format."); - - int fileCount = manifest.ReadInt32(); - if (fileCount < 0 || fileCount > manifest.BaseStream.Length - manifest.BaseStream.Position) - throw new InvalidDataException("Invalid ILLink cache file count."); - - var files = new List(); - var paths = new HashSet(RuntimeInformation.IsOSPlatform(OSPlatform.Windows) ? StringComparer.OrdinalIgnoreCase : StringComparer.Ordinal); - string outputsDirectory = Path.Combine(entryDirectory, "outputs"); - for (int i = 0; i < fileCount; i++) - { - string relativePath = manifest.ReadString(); - ValidateRelativePath(relativePath); - if (!paths.Add(relativePath)) - throw new InvalidDataException("Invalid ILLink cache output inventory."); - - long length = manifest.ReadInt64(); - byte[] expectedHash = manifest.ReadBytes(32); - string cachedFile = Path.Combine(outputsDirectory, relativePath); - - if (expectedHash.Length != 32 || new FileInfo(cachedFile).Length != length || - !expectedHash.SequenceEqual(HashFile(cachedFile))) - throw new InvalidDataException($"Invalid ILLink cache output: '{relativePath}'."); - - files.Add(relativePath); - } - - if (manifest.BaseStream.Position != manifest.BaseStream.Length) - throw new InvalidDataException("Unexpected data in ILLink cache manifest."); - - return files; - } - - private static void ValidateRelativePath(string path) - { - if (string.IsNullOrEmpty(path) || Path.IsPathRooted(path)) - throw new InvalidDataException("Invalid ILLink cache output path."); - - foreach (string component in path.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar)) - { - if (component is "" or "." or ".." || component.IndexOfAny(Path.GetInvalidFileNameChars()) >= 0 || - (RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && component[component.Length - 1] is '.' or ' ')) - throw new InvalidDataException("Invalid ILLink cache output path."); - } - } - - private static byte[] HashFile(string path) - { - using var sha256 = SHA256.Create(); - using var stream = File.OpenRead(path); - return sha256.ComputeHash(stream); - } - - private static bool IsCacheFailure(Exception ex) => ex is IOException or UnauthorizedAccessException or InvalidDataException or FormatException; -} diff --git a/src/tools/illink/src/ILLink.Tasks/LinkTask.cs b/src/tools/illink/src/ILLink.Tasks/LinkTask.cs index 0f43e75a857db0..9b9584dae6fde2 100644 --- a/src/tools/illink/src/ILLink.Tasks/LinkTask.cs +++ b/src/tools/illink/src/ILLink.Tasks/LinkTask.cs @@ -3,22 +3,23 @@ using System; using System.Collections.Generic; -using System.Globalization; using System.IO; using System.Linq; using System.Reflection; using System.Reflection.Metadata; using System.Reflection.PortableExecutable; -using System.Runtime.InteropServices; -using System.Security.Cryptography; using System.Text; using Microsoft.Build.Framework; using Microsoft.Build.Utilities; +using Microsoft.NET.Build.Tasks; namespace ILLink.Tasks { public class ILLink : ToolTask { + private static readonly InvocationCacheConfiguration s_cacheConfiguration = + new("ILLink", "ILLINK_EXPERIMENTAL_CACHE", "ILLINK_EXPERIMENTAL_CACHE_PATH"); + /// /// Paths to the assembly files that should be considered as /// input to the ILLink. @@ -269,22 +270,17 @@ private string DotNetPath protected override string GenerateFullPathToTool() => DotNetPath; protected override int ExecuteTool(string pathToTool, string responseFileCommands, string commandLineCommands) - { - DateTime outputTimestampUtc = DateTime.UtcNow; - ILLinkCache cache = ILLinkCache.TryCreateFromEnvironment(Log); - string inputHash = null; - if (cache is not null && !TryComputeCacheKey(pathToTool, commandLineCommands, responseFileCommands, out inputHash)) - cache = null; - - if (cache is not null && cache.TryRestore(inputHash, OutputDirectory.ItemSpec, outputTimestampUtc)) - return 0; + => InvocationCache.ExecuteCached(Log, s_cacheConfiguration, + () => DescribeInvocation(pathToTool, responseFileCommands, commandLineCommands), + () => ExecuteNormally(pathToTool, responseFileCommands, commandLineCommands)); + private int ExecuteNormally(string pathToTool, string responseFileCommands, string commandLineCommands) + { try { string outputDirectory = Path.GetFullPath(OutputDirectory.ItemSpec); if (Directory.Exists(outputDirectory)) Directory.Delete(outputDirectory, recursive: true); - Directory.CreateDirectory(outputDirectory); } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException) @@ -293,21 +289,17 @@ protected override int ExecuteTool(string pathToTool, string responseFileCommand return -1; } - int exitCode = base.ExecuteTool(pathToTool, responseFileCommands, commandLineCommands); - if (exitCode == 0 && !Log.HasLoggedErrors && cache is not null) - cache.Store(inputHash, OutputDirectory.ItemSpec); - - return exitCode; + return base.ExecuteTool(pathToTool, responseFileCommands, commandLineCommands); } - private bool TryComputeCacheKey(string pathToTool, string commandLineCommands, string responseFileCommands, out string inputHash) +#nullable enable annotations + private InvocationDescription? DescribeInvocation(string pathToTool, string responseFileCommands, string commandLineCommands) { - inputHash = string.Empty; if (CustomSteps?.Length > 0 || CustomData?.Length > 0 || DumpDependencies || !string.IsNullOrEmpty(DependenciesFileFormat) || EnvironmentVariables?.Length > 0) { Log.LogMessage(MessageImportance.Low, "ILLink cache bypassed: custom steps/data, dependency dumps, and task environment overrides are not supported."); - return false; + return null; } try @@ -317,7 +309,7 @@ private bool TryComputeCacheKey(string pathToTool, string commandLineCommands, s if (!TryAddCacheableExtraArgsInputs(files, directories)) { Log.LogMessage(MessageImportance.Low, "ILLink cache bypassed: extra arguments are not supported."); - return false; + return null; } var assemblies = new HashSet(StringComparer.Ordinal); @@ -350,34 +342,11 @@ private bool TryComputeCacheKey(string pathToTool, string commandLineCommands, s AddFile(pathToTool); AddFile(ILLinkPath); - using var sha256 = SHA256.Create(); - using var description = new MemoryStream(); - using var writer = new BinaryWriter(description, Encoding.UTF8, leaveOpen: true); - writer.Write("ILLink task cache key v1"); - writer.Write(commandLineCommands); - writer.Write(responseFileCommands); - writer.Write(Environment.CurrentDirectory); - writer.Write(Path.GetFullPath(OutputDirectory.ItemSpec)); - writer.Write(RuntimeInformation.OSDescription); - writer.Write(RuntimeInformation.ProcessArchitecture.ToString()); - writer.Write(CultureInfo.CurrentCulture.Name); - writer.Write(CultureInfo.CurrentUICulture.Name); - writer.Write(files.Count); - foreach (string file in files) + return new InvocationDescription(InvocationInputs.FromFiles(files.ToArray()), new[] { - writer.Write(file); - using var input = File.OpenRead(file); - writer.Write(sha256.ComputeHash(input)); - } - - writer.Flush(); - description.Position = 0; -#if NET - inputHash = Convert.ToHexStringLower(sha256.ComputeHash(description)); -#else - inputHash = BitConverter.ToString(sha256.ComputeHash(description)).Replace("-", "").ToLowerInvariant(); -#endif - return true; + commandLineCommands, + responseFileCommands + }, InvocationOutputs.FromDirectory(OutputDirectory.ItemSpec)); void AddFile(string path) => files.Add(Path.GetFullPath(path)); @@ -420,11 +389,13 @@ void AddAssembly(string path) } catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or ArgumentException or NotSupportedException or BadImageFormatException or InvalidDataException) { - Log.LogMessage(MessageImportance.Low, $"ILLink cache bypassed: input identity could not be computed: {ex.Message}"); - return false; + Log.LogMessage(MessageImportance.Low, $"ILLink cache bypassed: invocation could not be described: {ex.Message}"); + return null; } } +#nullable restore annotations + private bool TryAddCacheableExtraArgsInputs(SortedSet files, SortedSet directories) { ReadOnlySpan arguments = ExtraArgs.AsSpan().Trim(); @@ -451,10 +422,15 @@ option is not ("--link-attributes" or "--substitutions" or "-d") || return false; } + string fullPath = Path.GetFullPath(path); if (option == "-d") - directories.Add(Path.GetFullPath(path)); + directories.Add(fullPath); else - files.Add(Path.GetFullPath(path)); + { + // Explicit XML arguments must be readable, not merely recorded as absent. + using FileStream input = File.OpenRead(fullPath); + files.Add(fullPath); + } } return true; diff --git a/src/tools/illink/test/ILLink.CacheTool.Tests/CachePurgeTests.cs b/src/tools/illink/test/ILLink.CacheTool.Tests/CachePurgeTests.cs deleted file mode 100644 index 14f15cfe20c7ac..00000000000000 --- a/src/tools/illink/test/ILLink.CacheTool.Tests/CachePurgeTests.cs +++ /dev/null @@ -1,245 +0,0 @@ -// Copyright (c) .NET Foundation and contributors. All rights reserved. -// Licensed under the MIT license. See LICENSE file in the project root for full license information. - -using System; -using System.Globalization; -using System.IO; -using ILLink.Tasks; -using Xunit; - -namespace ILLink.CacheTool.Tests; - -public class CachePurgeTests -{ - [Theory] - [InlineData(-1, true)] - [InlineData(0, false)] - [InlineData(1, false)] - public void PurgeUsesLastUsedContentsAndExclusiveCutoff(int ticks, bool deleted) - { - using var cache = new TestCache(); - string entry = cache.AddEntry('a', TestCache.Cutoff.AddTicks(ticks)); - File.SetLastWriteTimeUtc(Path.Combine(entry, ILLinkCacheEntry.LastUsedFileName), DateTime.UtcNow); - - Assert.Equal(0, cache.Run()); - - Assert.Equal(!deleted, Directory.Exists(entry)); - Assert.Equal($"ILLink cache purge: Deleted: {(deleted ? 1 : 0)}, Kept: {(deleted ? 0 : 1)}, Errors: 0{Environment.NewLine}", - cache.Output.ToString()); - Assert.Empty(cache.Error.ToString()); - } - - [Fact] - public void PurgeOnlyVisitsPublishedEntriesInCurrentLayout() - { - using var cache = new TestCache(); - string old = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); - string recent = cache.AddEntry('b', TestCache.Cutoff.AddDays(1)); - foreach (string name in new[] { new string('a', 64) + ".unique.tmp", "unrelated", new string('E', 64), new string('c', 63) }) - { - Directory.CreateDirectory(Path.Combine(cache.Entries, name)); - } - string otherVersion = Path.Combine(cache.Root, "v2", new string('d', 64)); - Directory.CreateDirectory(otherVersion); - File.WriteAllText(Path.Combine(cache.Entries, "unrelated-file"), "keep"); - - Assert.Equal(0, cache.Run()); - - Assert.False(Directory.Exists(old)); - Assert.True(Directory.Exists(recent)); - Assert.Equal(5, Directory.GetDirectories(cache.Entries).Length); - Assert.True(Directory.Exists(otherVersion)); - Assert.Equal("keep", File.ReadAllText(Path.Combine(cache.Entries, "unrelated-file"))); - Assert.Contains("Deleted: 1, Kept: 1, Errors: 0", cache.Output.ToString()); - } - - [Theory] - [InlineData("missing", false)] - [InlineData("malformed", false)] - [InlineData("empty", false)] - [InlineData("directory", false)] - [InlineData("non-utc", false)] - [InlineData("missing", true)] - [InlineData("malformed", true)] - [InlineData("empty", true)] - [InlineData("directory", true)] - [InlineData("non-utc", true)] - public void InvalidMarkerUsesEntryCreationTime(string damage, bool delete) - { - using var cache = new TestCache(); - string damagedEntry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); - string validEntry = cache.AddEntry('b', TestCache.Cutoff.AddDays(-1)); - string marker = Path.Combine(damagedEntry, ILLinkCacheEntry.LastUsedFileName); - switch (damage) - { - case "missing": - File.Delete(marker); - break; - case "malformed": - File.WriteAllText(marker, "invalid timestamp"); - break; - case "empty": - File.WriteAllText(marker, ""); - break; - case "directory": - File.Delete(marker); - Directory.CreateDirectory(marker); - break; - case "non-utc": - File.WriteAllText(marker, TestCache.Cutoff.ToOffset(TimeSpan.FromHours(1)).ToString("O", CultureInfo.InvariantCulture)); - break; - } - - Assert.Equal(0, cache.Run(delete ? DateTimeOffset.UtcNow.AddMinutes(1) : TestCache.Cutoff)); - - Assert.Equal(!delete, Directory.Exists(damagedEntry)); - Assert.False(Directory.Exists(validEntry)); - Assert.Contains(damagedEntry, cache.Error.ToString()); - Assert.Contains("using directory creation time", cache.Error.ToString()); - Assert.Equal($"ILLink cache purge: Deleted: {(delete ? 2 : 1)}, Kept: {(delete ? 0 : 1)}, Errors: 0{Environment.NewLine}", - cache.Output.ToString()); - } - - [Fact] - public void MissingVersionDirectoryIsReported() - { - using var cache = new TestCache(); - Directory.Delete(cache.Entries); - - Assert.Equal(1, cache.Run()); - - Assert.Contains(cache.Entries, cache.Error.ToString()); - Assert.Contains("Errors: 1", cache.Output.ToString()); - Assert.False(Directory.Exists(cache.Entries)); - } - - [Fact] - public void EmptyCacheSucceeds() - { - using var cache = new TestCache(); - Assert.Equal(0, cache.Run()); - Assert.Contains("Deleted: 0, Kept: 0, Errors: 0", cache.Output.ToString()); - } - - [Theory] - [InlineData("2025-01-02T03:04:05Z")] - [InlineData("2025-01-02T03:04:05+00:00")] - [InlineData("2025-01-02T03:04:05.0000000+00:00")] - [InlineData("2025-01-02T03:04:05.0000000Z")] - public void CommandAcceptsExplicitUtcCutoff(string cutoff) - { - using var cache = new TestCache(); - string entry = cache.AddEntry('a', TestCache.Cutoff.AddTicks(-1)); - - Assert.Equal(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", cutoff }, cache.Output, cache.Error)); - - Assert.False(Directory.Exists(entry)); - Assert.Contains("Deleted: 1", cache.Output.ToString()); - Assert.Empty(cache.Error.ToString()); - } - - [Theory] - [InlineData("invalid")] - [InlineData("")] - [InlineData("2025-01-02")] - [InlineData("2025-01-02T03:04:05")] - [InlineData("2025-01-02T03:04:05+01:00")] - public void CommandRejectsInvalidCutoffWithoutDeleting(string cutoff) - { - using var cache = new TestCache(); - string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); - - Assert.NotEqual(0, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", cutoff }, cache.Output, cache.Error)); - - Assert.True(Directory.Exists(entry)); - Assert.Contains("--before", cache.Error.ToString()); - Assert.DoesNotContain("ILLink cache purge:", cache.Output.ToString()); - } - - [Theory] - [InlineData("missing-directory")] - [InlineData("missing-cutoff")] - [InlineData("missing-value")] - [InlineData("unknown-option")] - [InlineData("unknown-command")] - public void CommandRejectsIncompleteOrUnknownArguments(string scenario) - { - using var cache = new TestCache(); - string[] args = scenario switch - { - "missing-directory" => new[] { "purge", "--before", "2025-01-02T03:04:05Z" }, - "missing-cutoff" => new[] { "purge", "--cache-directory", cache.Root }, - "missing-value" => new[] { "purge", "--cache-directory", cache.Root, "--before" }, - "unknown-option" => new[] { "purge", "--cache-directory", cache.Root, "--before", "2025-01-02T03:04:05Z", "--force" }, - _ => new[] { "delete-all" } - }; - string entry = cache.AddEntry('a', TestCache.Cutoff.AddDays(-1)); - - Assert.NotEqual(0, Program.Run(args, cache.Output, cache.Error)); - - Assert.True(Directory.Exists(entry)); - Assert.NotEmpty(cache.Error.ToString()); - } - - [Fact] - public void CommandRejectsNonexistentDirectory() - { - using var cache = new TestCache(); - string missing = Path.Combine(cache.Root, "missing"); - - Assert.NotEqual(0, Program.Run(new[] { "purge", "--cache-directory", missing, "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); - - Assert.NotEmpty(cache.Error.ToString()); - Assert.False(Directory.Exists(missing)); - } - - [Fact] - public void CommandPropagatesMaintenanceFailure() - { - using var cache = new TestCache(); - Directory.Delete(cache.Entries); - - Assert.Equal(1, Program.Run(new[] { "purge", "--cache-directory", cache.Root, "--before", "2025-01-02T03:04:05Z" }, cache.Output, cache.Error)); - - Assert.Contains("Errors: 1", cache.Output.ToString()); - } - - [Fact] - public void HelpExplainsIdleCacheRequirement() - { - using var output = new StringWriter(); - using var error = new StringWriter(); - Assert.Equal(0, Program.Run(new[] { "purge", "--help" }, output, error)); - Assert.Contains("No builds may use this cache while purging.", output.ToString()); - Assert.Empty(error.ToString()); - } - - private sealed class TestCache : IDisposable - { - internal static readonly DateTimeOffset Cutoff = new(2025, 1, 2, 3, 4, 5, TimeSpan.Zero); - internal string Root { get; } = Path.Combine(Path.GetTempPath(), "illink-purge-tests-" + Guid.NewGuid().ToString("N")); - internal string Entries => Path.Combine(Root, ILLinkCacheEntry.VersionDirectory); - internal StringWriter Output { get; } = new(); - internal StringWriter Error { get; } = new(); - - internal TestCache() => Directory.CreateDirectory(Entries); - - internal string AddEntry(char key, DateTimeOffset lastUsed) - { - string entry = Path.Combine(Entries, new string(key, 64)); - Directory.CreateDirectory(Path.Combine(entry, "outputs")); - File.WriteAllText(Path.Combine(entry, "outputs", "app.dll"), "cached output"); - File.WriteAllText(Path.Combine(entry, ILLinkCacheEntry.LastUsedFileName), lastUsed.ToString("O", CultureInfo.InvariantCulture)); - return entry; - } - - internal int Run(DateTimeOffset? cutoff = null) => CachePurge.Run(Root, cutoff ?? Cutoff, Output, Error); - - public void Dispose() - { - Output.Dispose(); - Error.Dispose(); - Directory.Delete(Root, recursive: true); - } - } -} diff --git a/src/tools/illink/test/ILLink.CacheTool.Tests/ILLink.CacheTool.Tests.csproj b/src/tools/illink/test/ILLink.CacheTool.Tests/ILLink.CacheTool.Tests.csproj deleted file mode 100644 index 8822ac9bdd82f0..00000000000000 --- a/src/tools/illink/test/ILLink.CacheTool.Tests/ILLink.CacheTool.Tests.csproj +++ /dev/null @@ -1,5 +0,0 @@ - - - - - diff --git a/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.cs b/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.cs index 84135d1a807f87..db573a9cb68af1 100644 --- a/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.cs +++ b/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.cs @@ -3,18 +3,16 @@ using System; using System.Collections.Generic; -using System.Globalization; using System.IO; using System.Linq; using System.Reflection; using System.Reflection.Metadata; using System.Reflection.Metadata.Ecma335; using System.Reflection.PortableExecutable; -using System.Runtime.ExceptionServices; -using System.Threading; using Microsoft.Build.Framework; using Microsoft.Build.Utilities; using Microsoft.DotNet.RemoteExecutor; +using Microsoft.NET.Build.Tasks; using Mono.Linker; using Xunit; @@ -957,6 +955,55 @@ public void TaskOutputPreparationFailureDoesNotRunTool() Assert.Empty(test.Task.Messages); } + [Theory] + [InlineData("equal-directory")] + [InlineData("cache-in-output")] + [InlineData("output-in-cache")] + public void TaskRejectsOverlappingCacheBeforePreparingOutputs(string overlap) + { + RemoteExecutor.Invoke(static value => + { + using var test = new OutputDirectoryTest(); + string input = Path.Combine(test.Root, "Input.dll"); + WriteTestAssembly(input); + var task = test.Task; + task.AssemblyPaths = new ITaskItem[] { new TaskItem(input, new Dictionary { { "TrimMode", "copy" } }) }; + task.RootAssemblyNames = new ITaskItem[] { new TaskItem("Input") }; + task.ExtraArgs = null; + string cacheDirectory = value switch + { + "equal-directory" => test.Output, + "cache-in-output" => Path.Combine(test.Output, "cache"), + "output-in-cache" => Path.Combine(test.Root, "cache"), + _ => throw new InvalidOperationException() + }; + string outputDirectory = value == "output-in-cache" + ? Path.Combine(cacheDirectory, "linked output") : test.Output; + task.OutputDirectory = new TaskItem(outputDirectory); + Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE", "true"); + Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE_PATH", cacheDirectory); + string source = Path.Combine(test.Root, "cached outputs"); + Directory.CreateDirectory(source); + File.Copy(input, Path.Combine(source, "Input.dll")); + var cache = new InvocationCache(cacheDirectory, task.Log, "ILLink"); + string key = new string('a', 64); + cache.Store(key, InvocationOutputs.FromDirectory(source)); + string cachedImage = Path.Combine(cacheDirectory, InvocationCacheEntry.VersionDirectory, key, "outputs", "Input.dll"); + Assert.Equal(File.ReadAllBytes(input), File.ReadAllBytes(cachedImage)); + Directory.CreateDirectory(outputDirectory); + string stale = Path.Combine(outputDirectory, "stale.dll"); + File.WriteAllText(stale, "untouched"); + + Assert.False(task.Execute()); + + Assert.Equal("untouched", File.ReadAllText(stale)); + Assert.Equal(File.ReadAllBytes(input), File.ReadAllBytes(cachedImage)); + Assert.Contains(test.BuildEngine.Errors, error => + error.Code == "MSB6001" && error.Message.Contains("The cache and output directories must not contain each other.")); + Assert.Empty(task.Messages); + }, overlap).Dispose(); + } + [Theory] [InlineData(null)] [InlineData("-o")] @@ -1022,538 +1069,6 @@ public void CacheHasNoTaskParameters() Assert.DoesNotContain(typeof(ILLink).GetProperties(), property => property.Name.Contains("Cache")); } - [Theory] - [InlineData(null, false, false)] - [InlineData("", false, false)] - [InlineData("false", false, false)] - [InlineData("FALSE", false, false)] - [InlineData("true", true, false)] - [InlineData("TrUe", true, false)] - [InlineData("1", false, true)] - [InlineData("invalid", false, true)] - public void CacheEnvironmentConfiguration(string setting, bool enabled, bool invalid) - { - RemoteExecutor.Invoke(static (value, expectedEnabled, expectedInvalid) => - { - Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE", value == "" ? null : value); - string directory = Path.Combine(Path.GetTempPath(), "illink-cache-" + Guid.NewGuid().ToString("N")); - Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE_PATH", directory); - var buildEngine = new MockBuildEngine(); - var task = new MockTask { BuildEngine = buildEngine }; - ILLinkCache cache = ILLinkCache.TryCreateFromEnvironment(task.Log); - - Assert.Equal(bool.Parse(expectedEnabled), cache is not null); - if (cache is not null) - Assert.Equal(directory, cache.CacheDirectory); - Assert.Equal(bool.Parse(expectedInvalid), buildEngine.Messages.Any(message => - message.Message.Contains("ILLINK_EXPERIMENTAL_CACHE must be 'true' or 'false'"))); - Assert.False(Directory.Exists(directory)); - }, setting ?? "", enabled.ToString(), invalid.ToString()).Dispose(); - } - - [Theory] - [InlineData(null)] - [InlineData("")] - [InlineData("true")] - [InlineData("false")] - [InlineData("cache directory")] - public void CacheEnvironmentDirectory(string directory) - { - RemoteExecutor.Invoke(static value => - { - string path = value == "" ? null : value; - Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE", "true"); - Environment.SetEnvironmentVariable("ILLINK_EXPERIMENTAL_CACHE_PATH", path); - var task = new MockTask { BuildEngine = new MockBuildEngine() }; - - ILLinkCache cache = ILLinkCache.TryCreateFromEnvironment(task.Log); - - Assert.NotNull(cache); - Assert.Equal(ILLinkCache.TryCreate(path, task.Log).CacheDirectory, cache.CacheDirectory); - }, directory ?? "").Dispose(); - } - - [Theory] - [InlineData(null)] - [InlineData("")] - public void CacheUsesPlatformDefaultDirectory(string cacheDirectory) - { - var task = new MockTask { BuildEngine = new MockBuildEngine() }; - string expectedDirectory; - if (OperatingSystem.IsWindows()) - { - expectedDirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "illink"); - } - else if (OperatingSystem.IsMacOS()) - { - expectedDirectory = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), "Library", "Caches", "illink"); - } - else - { - string cacheHome = Environment.GetEnvironmentVariable("XDG_CACHE_HOME"); - expectedDirectory = !string.IsNullOrEmpty(cacheHome) && Path.IsPathRooted(cacheHome) - ? Path.Combine(cacheHome, "illink") - : Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".cache", "illink"); - } - - ILLinkCache cache = ILLinkCache.TryCreate(cacheDirectory, task.Log); - - Assert.NotNull(cache); - Assert.Equal(Path.GetFullPath(expectedDirectory), cache.CacheDirectory); - } - - [Theory] - [InlineData(false)] - [InlineData(true)] - public void CacheDirectoryOverrideDoesNotCreateDirectory(bool absolutePath) - { - string directoryName = "illink-cache-" + Guid.NewGuid().ToString("N"); - string cacheDirectory = absolutePath ? Path.Combine(Path.GetTempPath(), directoryName) : directoryName; - var task = new MockTask { BuildEngine = new MockBuildEngine() }; - - ILLinkCache cache = ILLinkCache.TryCreate(cacheDirectory, task.Log); - - Assert.NotNull(cache); - Assert.Equal(Path.GetFullPath(cacheDirectory), cache.CacheDirectory); - Assert.False(Directory.Exists(cache.CacheDirectory)); - } - - [Fact] - public void InvalidCacheDirectoryBypassesCaching() - { - var buildEngine = new MockBuildEngine(); - var task = new MockTask { BuildEngine = buildEngine }; - - Assert.Null(ILLinkCache.TryCreate("invalid\0path", task.Log)); - Assert.Contains(buildEngine.Messages, message => - message.Message.Contains("could not be resolved")); - } - - [Theory] - [InlineData(false)] - [InlineData(true)] - public void CacheRoundTrip(bool includeSymbols) - { - using var test = new CacheTestDirectory(); - var files = new Dictionary - { - ["app.dll"] = "assembly", - ["app.dll.config"] = "configuration", - [Path.Combine("fr", "app.resources.dll")] = "satellite", - ["Link.semaphore"] = "ordinary file" - }; - if (includeSymbols) - files.Add("app.pdb", "symbols"); - - foreach (var file in files) - test.WriteSource(file.Key, file.Value); - - Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.False(Directory.Exists(test.Cache.CacheDirectory)); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - - Directory.CreateDirectory(test.Output); - File.WriteAllText(Path.Combine(test.Output, "app.dll"), "old assembly"); - File.WriteAllText(Path.Combine(test.Output, "Link.semaphore"), "existing semaphore"); - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - - foreach (var file in files) - { - string path = Path.Combine(test.Output, file.Key); - Assert.Equal(file.Value, File.ReadAllText(path)); - Assert.Equal(CacheTestDirectory.Timestamp, File.GetLastWriteTimeUtc(path)); - } - Assert.Equal("ordinary file", File.ReadAllText(Path.Combine(test.Output, "Link.semaphore"))); - Assert.Equal(files.Count, Directory.GetFiles(Path.Combine(test.Entry, "outputs"), "*", SearchOption.AllDirectories).Length); - Assert.Empty(Directory.GetDirectories(Path.GetDirectoryName(test.Entry), "*.tmp")); - - File.WriteAllText(Path.Combine(test.Output, "app.dll"), "modified restored output"); - Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Entry, "outputs", "app.dll"))); - } - - [Fact] - public void CacheSupportsEmptyOutput() - { - using var test = new CacheTestDirectory(); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.True(Directory.Exists(test.Output)); - Assert.Empty(Directory.GetFileSystemEntries(test.Output)); - } - - [Fact] - public void CacheRecordsLastUsedOnStoreAndHit() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - DateTimeOffset beforeStore = DateTimeOffset.UtcNow; - test.Cache.Store(CacheTestDirectory.Key, test.Source); - string marker = Path.Combine(test.Entry, ILLinkCacheEntry.LastUsedFileName); - Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), - beforeStore, DateTimeOffset.UtcNow); - File.WriteAllText(marker, DateTimeOffset.MinValue.ToString("O", CultureInfo.InvariantCulture)); - - DateTimeOffset beforeRestore = DateTimeOffset.UtcNow; - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - - Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), - beforeRestore, DateTimeOffset.UtcNow); - Assert.False(File.Exists(Path.Combine(test.Output, ILLinkCacheEntry.LastUsedFileName))); - Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); - } - - [Fact] - public void CacheMarkerUpdateFailureDoesNotFailRestore() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - string marker = Path.Combine(test.Entry, ILLinkCacheEntry.LastUsedFileName); - File.Delete(marker); - Directory.CreateDirectory(marker); - - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - - Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Output, "app.dll"))); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("last-used update failed")); - Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); - Assert.Empty(test.BuildEngine.Errors); - } - - [Fact] - public void FailedRestoreDoesNotRefreshUsage() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - string marker = Path.Combine(test.Entry, ILLinkCacheEntry.LastUsedFileName); - string oldTimestamp = DateTimeOffset.MinValue.ToString("O", CultureInfo.InvariantCulture); - File.WriteAllText(marker, oldTimestamp); - File.WriteAllText(Path.Combine(test.Entry, "outputs", "app.dll"), "corrupt"); - - Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - - Assert.Equal(oldTimestamp, File.ReadAllText(marker)); - } - - [Fact] - public void ParallelProcessesRefreshUsageWithoutCorruptingMarker() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - DateTimeOffset beforeRestore = DateTimeOffset.UtcNow; - using (StartReader(test, 0)) - using (StartReader(test, 1)) - { - Assert.True(SpinWait.SpinUntil(() => Directory.GetFiles(test.Root, "*.ready").Length == 2, TimeSpan.FromSeconds(30))); - File.WriteAllText(Path.Combine(test.Root, "start"), ""); - } - - string marker = Path.Combine(test.Entry, ILLinkCacheEntry.LastUsedFileName); - Assert.InRange(DateTimeOffset.ParseExact(File.ReadAllText(marker), "O", CultureInfo.InvariantCulture), - beforeRestore, DateTimeOffset.UtcNow); - Assert.Empty(Directory.GetFiles(test.Entry, "*.tmp")); - - static RemoteInvokeHandle StartReader(CacheTestDirectory test, int index) => - RemoteExecutor.Invoke(static (cacheDirectory, output, start) => - { - var engine = new MockBuildEngine(); - var task = new MockTask { BuildEngine = engine }; - ILLinkCache cache = ILLinkCache.TryCreate(cacheDirectory, task.Log); - File.WriteAllText(output + ".ready", ""); - Assert.True(SpinWait.SpinUntil(() => File.Exists(start), TimeSpan.FromSeconds(30))); - for (int attempt = 0; attempt < 20; attempt++) - Assert.True(cache.TryRestore(CacheTestDirectory.Key, output, CacheTestDirectory.Timestamp)); - Assert.Empty(engine.Errors); - }, test.Cache.CacheDirectory, Path.Combine(test.Root, "reader-" + index), Path.Combine(test.Root, "start")); - } - - [Fact] - public void CacheDoesNotReplaceExistingEntry() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "first"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - test.WriteSource("app.dll", "second"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.Equal("first", File.ReadAllText(Path.Combine(test.Output, "app.dll"))); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("already exists")); - Assert.Empty(Directory.GetDirectories(Path.GetDirectoryName(test.Entry), "*.tmp")); - } - - [Theory] - [InlineData("missing-manifest")] - [InlineData("version")] - [InlineData("truncated")] - [InlineData("missing-file")] - [InlineData("corrupt-file")] - [InlineData("escaping-path")] - public void CacheRejectsInvalidEntryBeforeCopying(string damage) - { - using var test = new CacheTestDirectory(); - test.WriteSource("a.dll", "first"); - test.WriteSource("b.dll", "second"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - string manifest = Path.Combine(test.Entry, "manifest"); - string cachedFile = Path.Combine(test.Entry, "outputs", "b.dll"); - switch (damage) - { - case "missing-manifest": - File.Delete(manifest); - break; - case "version": - using (var writer = new BinaryWriter(File.OpenWrite(manifest))) - writer.Write(-1); - break; - case "truncated": - File.WriteAllBytes(manifest, new byte[] { 1 }); - break; - case "missing-file": - File.Delete(cachedFile); - break; - case "corrupt-file": - File.WriteAllText(cachedFile, "broken"); - break; - case "escaping-path": - using (var writer = new BinaryWriter(File.Create(manifest))) - { - writer.Write(1); - writer.Write(1); - writer.Write("../escaped.dll"); - } - break; - } - - Directory.CreateDirectory(test.Output); - string existingOutput = Path.Combine(test.Output, "a.dll"); - File.WriteAllText(existingOutput, "unchanged"); - Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.Equal("unchanged", File.ReadAllText(existingOutput)); - Assert.False(File.Exists(Path.Combine(test.Output, "b.dll"))); - Assert.False(File.Exists(Path.Combine(test.Root, "escaped.dll"))); - Assert.True(Directory.Exists(test.Entry)); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("restore failed")); - } - - [Fact] - public void CacheRestoreReplacesExistingDirectory() - { - using var test = new CacheTestDirectory(); - test.WriteSource("a.dll", "new assembly"); - test.WriteSource("b.dll", "other assembly"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - Directory.CreateDirectory(Path.Combine(test.Output, "b.dll")); - File.WriteAllText(Path.Combine(test.Output, "a.dll"), "old assembly"); - - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.Equal("new assembly", File.ReadAllText(Path.Combine(test.Output, "a.dll"))); - Assert.Equal("other assembly", File.ReadAllText(Path.Combine(test.Output, "b.dll"))); - } - - [Theory] - [InlineData(false)] - [InlineData(true)] - public void CacheStoreFailureIsBestEffort(bool failPublication) - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - if (failPublication) - { - Directory.CreateDirectory(Path.GetDirectoryName(test.Entry)); - File.WriteAllText(test.Entry, "blocks publication"); - } - else - { - File.WriteAllText(test.Cache.CacheDirectory, "blocks cache creation"); - } - - test.Cache.Store(CacheTestDirectory.Key, test.Source); - - Assert.Equal("assembly", File.ReadAllText(Path.Combine(test.Source, "app.dll"))); - Assert.False(Directory.Exists(test.Entry)); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("store failed")); - if (failPublication) - { - Assert.Equal("blocks publication", File.ReadAllText(test.Entry)); - Assert.Empty(Directory.GetDirectories(Path.GetDirectoryName(test.Entry), "*.tmp")); - } - } - - [Fact] - public void CacheSkipsContendedEntryButNotOtherKeys() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - using var mutex = new Mutex(false, test.MutexName); - Assert.True(mutex.WaitOne(0)); - try - { - RunOnOtherThread(() => test.Cache.Store(CacheTestDirectory.Key, test.Source)); - Assert.False(Directory.Exists(test.Cache.CacheDirectory)); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("another writer")); - - string otherKey = new string('b', 64); - RunOnOtherThread(() => test.Cache.Store(otherKey, test.Source)); - Assert.True(test.Cache.TryRestore(otherKey, test.Output, CacheTestDirectory.Timestamp)); - } - finally - { - mutex.ReleaseMutex(); - } - - test.Cache.Store(CacheTestDirectory.Key, test.Source); - Assert.True(Directory.Exists(test.Entry)); - } - - [Fact] - public void CacheReadsPublishedEntryWithoutWriterLock() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - using var mutex = new Mutex(false, test.MutexName); - Assert.True(mutex.WaitOne(0)); - try - { - RunOnOtherThread(() => - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp))); - } - finally - { - mutex.ReleaseMutex(); - } - } - - [Fact] - public void CacheSupportsConcurrentWritersWithDifferentOutputs() - { - using var test = new CacheTestDirectory(); - string contents = new string('a', 1024 * 1024); - System.Threading.Tasks.Parallel.For(0, 8, index => - { - string source = Path.Combine(test.Root, "source-" + index); - Directory.CreateDirectory(source); - File.WriteAllText(Path.Combine(source, "app.dll"), contents); - File.WriteAllText(Path.Combine(source, "app.pdb"), "symbols"); - var task = new MockTask { BuildEngine = new MockBuildEngine() }; - ILLinkCache cache = ILLinkCache.TryCreate(test.Cache.CacheDirectory, task.Log); - - cache.Store(CacheTestDirectory.Key, source); - }); - - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.Equal(contents, File.ReadAllText(Path.Combine(test.Output, "app.dll"))); - Assert.Equal("symbols", File.ReadAllText(Path.Combine(test.Output, "app.pdb"))); - Assert.Single(Directory.GetDirectories(Path.GetDirectoryName(test.Entry))); - } - - [Fact] - public void CacheIgnoresOtherWritersStagingDirectories() - { - using var test = new CacheTestDirectory(); - string stagingDirectory = test.Entry + ".orphan.tmp"; - Directory.CreateDirectory(stagingDirectory); - string marker = Path.Combine(stagingDirectory, "partial"); - File.WriteAllText(marker, "other attempt"); - - Assert.False(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - test.Cache.Store(CacheTestDirectory.Key, test.Source); - - Assert.Equal("other attempt", File.ReadAllText(marker)); - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - } - - [Theory] - [InlineData(false)] - [InlineData(true)] - public void CacheRejectsOverlappingDirectories(bool cacheContainsOutput) - { - using var test = new CacheTestDirectory(); - string output = cacheContainsOutput ? Path.Combine(test.Cache.CacheDirectory, "output") : test.Root; - - Assert.Throws(() => test.Cache.Store(CacheTestDirectory.Key, output)); - Assert.Throws(() => test.Cache.TryRestore(CacheTestDirectory.Key, output, CacheTestDirectory.Timestamp)); - Assert.False(Directory.Exists(test.Cache.CacheDirectory)); - } - - [Fact] - public void CacheRecoversAbandonedWriterLock() - { - using var test = new CacheTestDirectory(); - test.WriteSource("app.dll", "assembly"); - using var mutex = new Mutex(false, test.MutexName); - var owner = new Thread(() => mutex.WaitOne()); - owner.Start(); - owner.Join(); - - test.Cache.Store(CacheTestDirectory.Key, test.Source); - - Assert.True(test.Cache.TryRestore(CacheTestDirectory.Key, test.Output, CacheTestDirectory.Timestamp)); - Assert.Contains(test.BuildEngine.Messages, message => message.Message.Contains("abandoned")); - } - - [Theory] - [InlineData("")] - [InlineData("../entry")] - [InlineData("not-a-hash")] - public void CacheRejectsInvalidInputHash(string inputHash) - { - using var test = new CacheTestDirectory(); - Assert.Throws(() => test.Cache.Store(inputHash, test.Source)); - Assert.Throws(() => test.Cache.TryRestore(inputHash, test.Output, CacheTestDirectory.Timestamp)); - Assert.False(Directory.Exists(test.Cache.CacheDirectory)); - } - - private static void RunOnOtherThread(Action action) - { - ExceptionDispatchInfo exception = null; - var thread = new Thread(() => - { - try - { - action(); - } - catch (Exception ex) - { - exception = ExceptionDispatchInfo.Capture(ex); - } - }); - thread.Start(); - thread.Join(); - exception?.Throw(); - } - - private sealed class CacheTestDirectory : IDisposable - { - internal static readonly string Key = new string('a', 64); - internal static readonly DateTime Timestamp = new DateTime(2025, 1, 2, 3, 4, 5, DateTimeKind.Utc); - internal string Root { get; } = Path.Combine(Path.GetTempPath(), "illink-cache-tests-" + Guid.NewGuid().ToString("N")); - internal string Source => Path.Combine(Root, "source"); - internal string Output => Path.Combine(Root, "output"); - internal string Entry => Path.Combine(Cache.CacheDirectory, "v1", Key); - internal MockBuildEngine BuildEngine { get; } = new(); - internal ILLinkCache Cache { get; } - internal string MutexName => (string)typeof(ILLinkCache).GetMethod("GetWriterMutexName", BindingFlags.NonPublic | BindingFlags.Instance).Invoke(Cache, new object[] { Key }); - - internal CacheTestDirectory() - { - Directory.CreateDirectory(Source); - var task = new MockTask { BuildEngine = BuildEngine }; - Cache = ILLinkCache.TryCreate(Path.Combine(Root, "cache"), task.Log); - } - - internal void WriteSource(string relativePath, string contents) - { - string path = Path.Combine(Source, relativePath); - Directory.CreateDirectory(Path.GetDirectoryName(path)); - File.WriteAllText(path, contents); - } - - public void Dispose() => Directory.Delete(Root, recursive: true); - } - [Theory] [InlineData(false)] [InlineData(true)] @@ -1742,7 +1257,7 @@ public void CacheKeyBypassesUnmodeledSearchDirectoryInputs(string reason) Assert.False(test.Task.TryGetCacheKey(host, out string key)); Assert.Empty(key); Assert.Contains(test.BuildEngine.Messages, message => - message.Message.StartsWith("ILLink cache bypassed: input identity could not be computed:")); + message.Message.StartsWith("ILLink cache bypassed: invocation could not be described:")); } [Fact] @@ -1792,7 +1307,7 @@ public void CacheKeyBypassesMissingRuntimeXmlFile(string option) Assert.False(test.Task.TryGetCacheKey(host, out string key)); Assert.Empty(key); Assert.Contains(test.BuildEngine.Messages, message => - message.Message.StartsWith("ILLink cache bypassed: input identity could not be computed:")); + message.Message.StartsWith("ILLink cache bypassed: invocation could not be described:")); } [Theory] @@ -2012,7 +1527,6 @@ public void CacheKeyIgnoresHostDirectories(string relativePath) [InlineData("invalid-assembly")] [InlineData("metadata-less-assembly")] [InlineData("metadata-less-reference")] - [InlineData("missing-linker")] public void CacheKeyBypassesUnsupportedInputs(string reason) { using var test = new OutputDirectoryTest(); @@ -2052,9 +1566,6 @@ public void CacheKeyBypassesUnsupportedInputs(string reason) case "metadata-less-reference": WriteMetadataLessPE(test.Task.ReferenceAssemblyPaths[0].ItemSpec); break; - case "missing-linker": - File.Delete(test.Task.ILLinkPath); - break; } Assert.False(test.Task.TryGetCacheKey(host, out string key)); @@ -2062,6 +1573,26 @@ public void CacheKeyBypassesUnsupportedInputs(string reason) Assert.Contains(test.BuildEngine.Messages, message => message.Message.StartsWith("ILLink cache bypassed:")); } + [Theory] + [InlineData("linker")] + [InlineData("host")] + public void CacheKeyTracksToolFileAbsence(string tool) + { + using var test = new OutputDirectoryTest(); + string host = PrepareCacheKeyTest(test); + string path = tool == "linker" ? test.Task.ILLinkPath : host; + byte[] contents = File.ReadAllBytes(path); + Assert.True(test.Task.TryGetCacheKey(host, out string original)); + + File.Delete(path); + Assert.True(test.Task.TryGetCacheKey(host, out string missing)); + Assert.NotEqual(original, missing); + + File.WriteAllBytes(path, contents); + Assert.True(test.Task.TryGetCacheKey(host, out string restored)); + Assert.Equal(original, restored); + } + [Theory] [InlineData(false)] [InlineData(true)] @@ -2095,7 +1626,7 @@ public void TaskLinksWithUnusedMetadataLessReference(bool enableCache) if (bool.Parse(enabled)) { Assert.Contains(test.BuildEngine.Messages, message => - message.Message.StartsWith("ILLink cache bypassed: input identity could not be computed:") && + message.Message.StartsWith("ILLink cache bypassed: invocation could not be described:") && message.Message.Contains("no managed metadata")); } else @@ -2128,7 +1659,7 @@ public void TaskCachesAndInvalidatesOutputs(bool corruptEntry) Assert.True(task.Execute()); Assert.Contains(test.BuildEngine.Messages, message => message.Message.StartsWith("ILLink cache stored:")); - string entry = Assert.Single(Directory.GetDirectories(Path.Combine(cacheDirectory, "v1"))); + string entry = Assert.Single(Directory.GetDirectories(Path.Combine(cacheDirectory, InvocationCacheEntry.VersionDirectory))); string stale = Path.Combine(test.Output, "stale.txt"); File.WriteAllText(stale, "not part of the cached directory"); File.Delete(Path.Combine(test.Output, "Input.dll")); @@ -2160,7 +1691,7 @@ public void TaskCachesAndInvalidatesOutputs(bool corruptEntry) message.Message.StartsWith(corruptEntry ? "ILLink cache restore failed" : "ILLink cache miss:")); Assert.False(File.Exists(stale)); Assert.Equal(File.ReadAllBytes(input), File.ReadAllBytes(Path.Combine(test.Output, "Input.dll"))); - Assert.Equal(corruptEntry ? 1 : 2, Directory.GetDirectories(Path.Combine(cacheDirectory, "v1")).Length); + Assert.Equal(corruptEntry ? 1 : 2, Directory.GetDirectories(Path.Combine(cacheDirectory, InvocationCacheEntry.VersionDirectory)).Length); }, corruptEntry.ToString()).Dispose(); } diff --git a/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.csproj b/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.csproj index 132b8028e0fe19..c26ce455b729bd 100644 --- a/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.csproj +++ b/src/tools/illink/test/ILLink.Tasks.Tests/ILLink.Tasks.Tests.csproj @@ -1,6 +1,7 @@ + diff --git a/src/tools/illink/test/ILLink.Tasks.Tests/Mock.cs b/src/tools/illink/test/ILLink.Tasks.Tests/Mock.cs index ea495b045162dc..846927db67ce7c 100644 --- a/src/tools/illink/test/ILLink.Tasks.Tests/Mock.cs +++ b/src/tools/illink/test/ILLink.Tasks.Tests/Mock.cs @@ -8,6 +8,7 @@ using System.Linq; using System.Reflection; using Microsoft.Build.Framework; +using Microsoft.NET.Build.Tasks; using Mono.Linker; using Mono.Linker.Steps; @@ -27,13 +28,12 @@ public bool TryGetCacheKey(string pathToTool, out string key) commandLine = string.IsNullOrEmpty(commandLine) ? string.Empty : " " + commandLine; object[] arguments = { pathToTool, - AdjustCommandsForOperatingSystem(commandLine), AdjustCommandsForOperatingSystem(GenerateResponseFileCommands()), - null + AdjustCommandsForOperatingSystem(commandLine) }; - bool result = (bool)typeof(ILLink).GetMethod("TryComputeCacheKey", BindingFlags.Instance | BindingFlags.NonPublic).Invoke(this, arguments); - key = (string)arguments[3]; - return result; + var invocation = (InvocationDescription)typeof(ILLink).GetMethod("DescribeInvocation", BindingFlags.Instance | BindingFlags.NonPublic).Invoke(this, arguments); + key = string.Empty; + return invocation is not null && InvocationCache.TryComputeKey(Log, "ILLink", invocation, out key); } public MockTask()