Add AsyncBarrier.SignalAndWait(CancellationToken) overload

[Andrew Arnott (AArnott)]

Also optimize the pre-existing code.
The new overload returns ValueTask to leave open the option of optimizing to zero-allocation use via IValueTaskSource.

Closes #1329

[Lifeng Lu (lifengl)]

The PR looks good to me. It is interesting that we defined the signature of the new method to return ValueTask, instead of Task, which seems to make two overloads to have different return types, while it doesn't really save any memory allocation in this case, because we either return Task.CompletedTask or a TaskCompletionSource is always created in the implementation. Do we really have a strong reason to adopt ValueTask to make the two overloads different here?

[Andrew Arnott (AArnott)]

Do we really have a strong reason to adopt ValueTask to make the two overloads different here?

Yielding Task returning methods must allocate a new Task object on every invocation. ValueTask does not have this restriction. This is how .NET has achieved amortized alloc-free yielding awaits. They use ValueTask for many of their new APIs.
Given this is an async primitive (albeit a very uncommon one), it seems prudent to prepare our API to allow for alloc-free patterns wherever we have the opportunity.
You're right that our implementation isn't alloc-free in the yielding case today. But we could fix that as an implementation detail in the future if we needed to, but only if the return type we choose now supports it.