Lossy WebP output differs between the SIMD and scalar paths: the vectorized quantizer skips luma AC sharpening #3246

Prerequisites

• I have written a descriptive issue title
• I have verified that I am running the latest version of ImageSharp
• I have verified if the problem exist in both DEBUG and RELEASE mode
• I have searched open and closed issues to ensure it has not already been reported

ImageSharp version

main @ a73c43d (also present in v4.1.3, same code in QuantizeBlock)

Other ImageSharp packages and versions

None

Environment (Operating system, version and so on)

Windows 11 Pro 10.0.22631, x64

.NET Framework version

.NET 10 (net10.0)

Description

The lossy WebP encoder gives a different bitstream for the same image and options, depending on the CPU instruction set that is used.

QuantEnc.QuantizeBlock has three paths: AVX2, SSE4.1 and scalar. The scalar path quantizes abs(in) + sharpen, as libwebp does. This adds the frequency sharpening bias to the luma AC coefficients. The two vector paths compute that sum but do not use the result:

Avx2.Add(coeff0.AsInt16(), sharpen0);   // result discarded
Sse2.Add(coeff0.AsInt16(), sharpen0);   // result discarded
Sse2.Add(coeff8.AsInt16(), sharpen8);   // result discarded

So on x64 the luma AC coefficients are quantized without the sharpening bias. The scalar path is used on ARM64 and when hardware intrinsics are disabled, so the same image gives a different file on x64 than on ARM64. This is also why Encode_Lossy_WithAlpha_Works and Encode_Lossy_WithAlphaUsingCompression_Works need different expected file sizes per architecture (64,020 B on x64 and 64,060 B on ARM64, and 16,200 B and 16,240 B). The test comment says the difference comes from floating-point differences. With DOTNET_EnableHWIntrinsic=0, x64 also gives 64,060 B and 16,240 B.

Expected: all three paths give the same quantized levels, so the encoded file does not depend on the CPU.

A fix is proposed in #3234.

Steps to Reproduce

1. Create a console app on an x64 machine with AVX2 that references ImageSharp, with this Program.cs:

   using System.Security.Cryptography;
   using SixLabors.ImageSharp;
   using SixLabors.ImageSharp.Formats.Webp;
   using SixLabors.ImageSharp.PixelFormats;
   
   using Image<Rgb24> image = new(128, 128);
   image.ProcessPixelRows(accessor =>
   {
       for (int y = 0; y < accessor.Height; y++)
       {
           Span<Rgb24> row = accessor.GetRowSpan(y);
           for (int x = 0; x < row.Length; x++)
           {
               int n = ((x * 7919) ^ (y * 104729)) & 63;
               row[x] = new Rgb24((byte)((x * 2) + n), (byte)((y * 2) + n), (byte)(((x + y) & 255) ^ n));
           }
       }
   });
   
   using MemoryStream ms = new();
   image.SaveAsWebp(ms, new WebpEncoder { FileFormat = WebpFileFormatType.Lossy });
   Console.WriteLine($"{ms.Length} bytes, SHA-256 {Convert.ToHexString(SHA256.HashData(ms.ToArray()))}");

2. Run it with the default settings (AVX2 path).

3. Run it again in a new process with DOTNET_EnableAVX2=0 (SSE4.1 path).

4. Run it again in a new process with DOTNET_EnableHWIntrinsic=0 (scalar path, as on ARM64).

   dotnet run -c Release
   $env:DOTNET_EnableAVX2 = "0"; dotnet run -c Release; Remove-Item Env:DOTNET_EnableAVX2
   $env:DOTNET_EnableHWIntrinsic = "0"; dotnet run -c Release; Remove-Item Env:DOTNET_EnableHWIntrinsic

Actual result: the two vector paths give the same file, and the scalar path gives a different file.

default:                     6544 bytes, SHA-256 04DAB06474190B59811459F43728329EAC8AD3415B3280D566C9104FEFD6BDBC
DOTNET_EnableAVX2=0:         6544 bytes, SHA-256 04DAB06474190B59811459F43728329EAC8AD3415B3280D566C9104FEFD6BDBC
DOTNET_EnableHWIntrinsic=0:  6570 bytes, SHA-256 E34E34900B013BAC2047BB61ADDAEB7EA167CC924523B34F542F2206837AF4F2

Expected result: all three runs print the same size and hash. The scalar result is the correct one, because it applies the sharpening bias.

Images

No images needed; the repro builds the input in memory.