Box-unbox optimization swallows InvalidCastException on different signs #71632

Description

Values with different signs are casted with no exception while roundtripping through object.

Reproduction Steps

[MethodImpl(MethodImplOptions.NoInlining)]
private static sbyte Problem(byte a)
{
    object box = a;
    Use(ref box);
    return (sbyte)box;
}

[MethodImpl(MethodImplOptions.NoInlining)]
private static sbyte Problem2(byte a)
{
    object box = a;
    return (sbyte)box;
}

public static void Use<T>(ref T arg) { }

Expected behavior

Both methods throw InvalidCastException.

Actual behavior

Problem2 doesn't throw it.

Regression?

Yes, introduced probably in Box-unbox enum optimization introduced in #70167.

Known Workarounds

No response

Configuration

.Net 7 builds off main branch, Preview 5 doesn't reproduce it.

Other information

It's probably caused by getTypeForPrimitiveValueClass treating types of the same size as the same here:
https://github.com/dotnet/runtime/blob/main/src/coreclr/vm/jitinterface.cpp#L3974