Add static hash helper methods

@GrabYourPitchforks

Edit by @GrabYourPitchforks on 20 Jan 2020: Formal API proposal written at https://github.com/dotnet/corefx/issues/9369#issuecomment-576445733.

Computing a hash code currently requires this code:

        byte[] hashBytes;
        using (var hashAlgo = SHA256.Create())
            hashBytes = hashAlgo.ComputeHash(bytes);

Or the more awkward HashAlgorithm.Create("SHA256").

This code is alright. It's not the end of the world. But I think it should be slimmer than that:

var hashBytes = SHA256.ComputeHash(bytes);

Benefits:

This is more terse.
It's an expression as opposed to a statement. That makes it more composable. It can be a subexpression for example in a LINQ query.
No way to forget resource cleanup or mess up code quality otherwise.
SHA256.ComputeHash can look at it's input size and dynamically pick the fastest implementation. I found the following to be optimal through testing on Windows x64: estimatedDataLength <= 512 ? new SHA1Managed() : HashAlgorithm.Create("SHA1"). Apparently, using the Windows crypto API has quite some per-hash cost.

I request that static helper method be added to the framework. This seems like an attractive case for a community contribution.

Proposal:

Using this pattern on all of the HashAlgorithm-derived types that are not KeyedHashAlgorithm-derived types (MD5, SHA1, SHA256, SHA384, SHA512):

public partial class SHA256
{
    public static byte[] Hash(byte[] source) => throw null;
    public static byte[] Hash(ReadOnlySpan<byte> source) => throw null;
    public static int Hash(ReadOnlySpan<byte> source, ReadOnlySpan<byte> destination) => throw null;
    public static bool TryHash(ReadOnlySpan<byte> source, ReadOnlySpan<byte> destination, out int bytesWritten) => throw null;
}

@GSPP

Thanks @GSPP! @bartonjs take a look and tag this as appropriate.

There are both good and bad sides to this idea.

The succinctness, composability, and reduced strain on the finalizer are all good. But "mess up code quality" is hard to gauge... if you're hashing a lot then it's better to hold the hash object across calls to avoid the setup/teardown overhead.

If we were to make it a static method then we need to make it thread safe (by the general rules of the framework), which means one of

Writing it as a using statement, each call doing setup/teardown
Writing it with thread-local statics, holding each object forever
Writing it with a lock statement
Writing it using Interlocked.Exchange to swap it in and out of a static field, leaving one hasher (per algorithm) open forever.

None of those are hard, or necessarily bad, but they all have different tradeoffs; and I'm not sure we'd be able to determine which one is best for the framework; other than guessing arbitrarily.

And, of course, there's an instance v static naming problem. The good name (ComputeHash) is already taken as a public instance method on a base class. We could redefine it with new, but I'm not sure it can really be done in a way that helps more than it confuses. (Or maybe we do this all the time and I've just never noticed).

(And as for the switching between the managed and native implementations, that could be done on the .NET Framework implementation, but in .NET Core we don't have a managed implementation).

So... given that we've broken IncrementalHash out into its own class, I think this definitely makes sense to have been the original implementation model. I'm just not sure that it can really be worked in to the existing API without creating more confusion than benefit.

(Maybe a new class which has each algorithm as a method?

public static class Hash
{
    public static byte[] SHA256(byte[] data);
    // And the other overloads

    ...
}

)

I like the new class idea.

There could be an overload for Stream and, more questionably, an overload for string, Encoding. Hashing strings is common and often done wrong because the encoding aspect is not understood. This overload would gently inform the user that the encoding is an important choice.

I did not think of sharing the object. I like that a lot! That would increase efficiency to nearly optimal levels.

I consider it bad practice to reuse crypto objects because that's error potential in very sensitive code. That's even more reason to add the static functions. If the framework does it correctly callers can rely on it and forget about playing caching tricks.

Now the advice can be: "If you want to hash anything, just use the Hash class." That's super simple guidance with almost no potential of misuse. I like that a lot.

@GSPP

I've always been a fan of adding convenience methods for the very common situation of having all the data you intend to hash (or do other crypto operations with). I'd rather not add more classes since we already have classes people know to use; adding static methods should generally be fine (maybe something like Sha256.Hash(byte[])?).

As @GSPP pointed out, as the framework, we can experiment with the policies you floated and find something that generally works reasonably well. If somebody finds a situation that isn't good for them, they're free to manage their own instances. Among your suggestions, I like attempting to do some kind of sharing for general efficiency. As long as we're not holding on to a ton of hashers, I think it's fine to have them open for a while... they don't use that many resources.

We need API proposal.

API proposal:

Let's add static methods to the individual hash classes, e.g.

var hashBytes = SHA256.ComputeHash(bytes);

I'd not add a new class Hash (or similar name) with usage like Hash.SHA256(...). Each hash type is logically isolated. User code might add more hash algorithms that would not be represented on this Hash class leading to confusion.

Libraries providing more hash algorithms can simply implement the static method pattern set forth by the framework.

We could cache hash algorithm instances. One issue here is that permanent resource usage might result from performing just a single or a few hash operations.

The cache could either be a ThreadStatic field or a global pool. I suspect that the overhead of any global pool would destroy too much performance.

Here is a sample implementation for a thread static solution:

internal static class HashAlgorithmCache<T>
{
 static T managedInstance;
 public static T GetManagedInstance(Func<T> factory) =>
  return managedInstance ?? (managedInstance = factory());

 static T unmanagedInstance;
 public static T GetUnmanagedInstance(Func<T> factory) =>
  return unmanagedInstance ?? (unmanagedInstance = factory());

 public static T GetInstance(
   int? expectedDataLength,
   Func<T> managedFactory,
   Func<T> unmanagedFactory)
 {
  bool useManaged = expectedDataLength == null || expectedDataLength.Value <= 512;
  return
   useManaged ? GetManagedInstance(managedFactory) :
   GetUnmanagedInstance(unmanagedFactory);
 }
}

class SHA256
{
 public static byte[] ComputeHash(byte[] input)
 {
   var expectedDataLength = input.Length;
   var hashAlgo = HashAlgorithmCache<SHA256>.GetInstance(
     expectedDataLength,
     () => new SHA256Managed(),
     () => HashAlgorithm.Create("SHA256"));
   return hashAlgo.ComputeHash(bytes);
 }
}

The main advantage is simplicity and performance. The main downside is potentially permanent resource usage from those cached algorithms. There are two instances per thread which can result in hundreds of objects.

An alternative here is to simply not cache anything and create a fresh instance each time.

There should be overloads for accepting Span<byte>, Stream and possibly pipelines if that is possible from a dependency standpoint.

On Win10+, the OS exposes a one-shot p/invoke API BCryptHash. You don't even need to open or cache an algorithm handle; you can pass the well-known constant BCRYPT_SHA256_ALG_HANDLE (for SHA256) in as the first parameter and let the OS take care of everything on your behalf. Pseudohandles for other hash routines are defined in bcrypt.h.

On other OSes, we could have a thread-static WeakReference<HashAlgorithm> if we're concerned about holding on to these objects for too long. That would allow the GC to reclaim them.

API proposal from the thread, more formally stated:

namespace System.Security.Cryptography
{
    public class SHA256 : HashAlgorithm
    {
        /*
         * new proposed static API
         * uses 'new' keyword since instance member HashAlgorithm.ComputeHash(byte[]) exists
         */
        public static new byte[] ComputeHash(byte[] data);

        /*
         * new proposed static API
         * computes the hash of the source data and writes it to the destination buffer.
         * returns number of bytes written; throws if destination too small.
         */
        public static int ComputeHash(ReadOnlySpan<byte> source, Span<byte> destination);
    }

    // above static methods also added to classes MD5, SHA1, SHA384, SHA512
}

As ComputeHash is an instance method, making a static method with the same name seems overly confusing (removed ready-for-review)

It's actually worse than confusing, it's a recompile-breaking change. Anyone with a strongly typed SHA256 that currently invokes the instance method will get a compile error:

Error	CS0176	Member 'SHA256.ComputeHash(byte[])' cannot be accessed with an instance reference; qualify it with a type name instead

@bartonjs

@bartonjs You have an alternative proposal that puts these as static members on a new static type, which would work around the issue. Even with the issues you point out it still seems like something we'd allow to come through API review since it's an actionable, concrete proposal.

The proposal for a new type named Hash? Since we've spanified things since then, I think it would be

public static class Hash
{
     public static byte[] SHA256(byte[] input) => throw null;
     public static int SHA256(ReadOnlySpan<byte> input, Span<byte> destination) => throw null;

     //repeat for MD5, SHA1, SHA384, SHA512
}

Morgan's suggestion was to use Hash as the method name for the one-shot, which would be

public partial class SHA256
{
    public static byte[] Hash(byte[] input) => throw null;
    public static int Hash(ReadOnlySpan<byte> input, ReadOnlySpan<byte> destination) => throw null;
}

The latter generalizes better (anyone who adds a hash algorithm in a NuGet package, or whatever, can follow that pattern, but they can't add static methods on our Hash accelerator type).

Having done a lot of work lately on patterns and practices, I think I prefer it as methods on the existing classes, as long as Hash (or some other non-breaking-change verb) seems discoverable enough.

Maybe HashBuffer? That would signal that a single buffer is being processed. This is in contrast to the other more general APIs (TransformBlock, ...).

Alternatives: HashBytes, GetHash.

Video

We discussed this again, and accepted HashData.

public partial class MD5
{
    public static byte[] HashData(byte[] source) => throw null;
    public static byte[] HashData(ReadOnlySpan<byte> source) => throw null;
    public static int HashData(ReadOnlySpan<byte> source, Span<byte> destination) => throw null;
    public static bool TryHashData(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten) => throw null;
}
public partial class SHA1
{
    public static byte[] HashData(byte[] source) => throw null;
    public static byte[] HashData(ReadOnlySpan<byte> source) => throw null;
    public static int HashData(ReadOnlySpan<byte> source, Span<byte> destination) => throw null;
    public static bool TryHashData(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten) => throw null;
}
public partial class SHA256
{
    public static byte[] HashData(byte[] source) => throw null;
    public static byte[] HashData(ReadOnlySpan<byte> source) => throw null;
    public static int HashData(ReadOnlySpan<byte> source, Span<byte> destination) => throw null;
    public static bool TryHashData(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten) => throw null;
}
public partial class SHA384
{
    public static byte[] HashData(byte[] source) => throw null;
    public static byte[] HashData(ReadOnlySpan<byte> source) => throw null;
    public static int HashData(ReadOnlySpan<byte> source, Span<byte> destination) => throw null;
    public static bool TryHashData(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten) => throw null;
}
public partial class SHA512
{
    public static byte[] HashData(byte[] source) => throw null;
    public static byte[] HashData(ReadOnlySpan<byte> source) => throw null;
    public static int HashData(ReadOnlySpan<byte> source, Span<byte> destination) => throw null;
    public static bool TryHashData(ReadOnlySpan<byte> source, Span<byte> destination, out int bytesWritten) => throw null;
}

I think I can squeeze this one in over the weekend.

Regarding the suggestion of "HashBytes": there are existing methods named (Try)HashData... admittedly, they're all protected; but there aren't any named "HashBytes", so "HashData" is more platform-consistent.

I think I can squeeze this one in over the weekend.

If you want to PAL through to native one-shots, that'd be best. If you want to go quicker with [ThreadStatic], that's fine too.

If you want to PAL through to native one-shots, that'd be best

That's the intention.

We don't really have a native PAL for CNG (do we?) and CNG's one-shot BCryptHash is Windows 10+, so maybe both?

We don't really have a native PAL for CNG (do we?)

Well, we have the PAL (split build for the OS), we just don't carry a native shim.

New APIs are in. Work left to be done:

Cache results of EVP_sha1, EVP_sha256, etc in HashAlgorithmToEvp to avoid a P/Invoke call.
Optimize CNG implementation to not allocate a stateful hash object.

Navigation Menu

Repository navigation

Add static hash helper methods #17590

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add static hash helper methods #17590

Description

Activity

joshfree commented on Jun 12, 2016

bartonjs commented on Jun 13, 2016

GSPP commented on Jun 13, 2016

morganbr commented on Jun 16, 2016

karelz commented on Nov 16, 2016

GSPP commented on Aug 12, 2018

GrabYourPitchforks commented on Feb 28, 2019

GrabYourPitchforks commented on Jan 20, 2020

bartonjs commented on Jan 20, 2020

bartonjs commented on Jan 20, 2020

GrabYourPitchforks commented on Jan 20, 2020

bartonjs commented on Jan 20, 2020

23 remaining items

GSPP commented on Jul 10, 2020

bartonjs commented on Jul 10, 2020

vcsjones commented on Jul 10, 2020

bartonjs commented on Jul 10, 2020

bartonjs commented on Jul 10, 2020

vcsjones commented on Jul 10, 2020

bartonjs commented on Jul 10, 2020

vcsjones commented on Jul 15, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions