Pixeldata not correct when reading certain JPEG #245
Description
Activity
Aha, didn't do a good enough check through the open issues then, my apologies for that :). Do you know the piece of code/class that is most likely to cause this issue? I could also do some investigation to see if I can find out anything.
Reacted by James Jackson-SouthYou're additional info is very important so don't worry. 😄 Tbh understanding jpeg eludes me. We're going wrong somewhere but I'm not sure where.
@devedse in the best case this might be caused by rounding inaccuracy in
Block8x8F.CopyColorsTo(). Adding0.5before casting to bytes might help here.In the worst case we need a deep refactor of the codebase to fix this; introducing an elegant management of Speed <-> Memory <-> Quality tradeoffs (see #192).
@antonfirsov I had a look at the rounding there and we probably should be adding
.5Fthere but that isn't the cause of our issue. I've previously checked theYCbCrconversion and that meets accuracy standards. (I used slow but high precision conversion to compare and our decoder was still incorrect) so it's not that either.The error must be elsewhere, I just don't know where that is. We're sometimes showing an error of +-3 per color component.
In the JpegDecoderCore I added a breakpoint if Pixel 5990, 3992 was being handled. I manually entered the YUV values in an online YUV to RGB converted and saw the same values as the ones that came out of your conversion function:
Website:
https://www.mikekohn.net/file_formats/yuv_rgb_converter.phpThe expected YUV values:
For this specific pixel it seems that:
Y: 20 instead of 21
U: 122 (Correct)
V: 140 instead of 139When I print out the surrounding values of the Y table for X values:
5985: 22
5986: 20
5987: 20
5988: 21
5989: 21
5990: 21 <---- Should be 20
5991: 21
5992: 19
5993: 20
5994: 20My next question would be, where does it fill the this.ycbcrImage.YChannel table?
@devedse Thanks for clarifying the conversion accuracy.
The channel is populated here. https://github.com/JimBobSquarePants/ImageSharp/blob/b165c54bb5c3a0c17cf6ce45f0f47fee9ff980fb/src/ImageSharp/Formats/Jpeg/Components/Decoder/JpegPixelArea.cs#L109
@Toxantron, I know, I just don't like the way they work. This works easier for me ^.^
Reacted by Thomas Fuchs and Anton FirszovDid some more debugging through the code to find out where the 21 comes from:
Output:
ValueSource: 21,36298 ValueAsByte: 21Not sure if it'll help you guys anything?
Reacted by Anton FirszovI also did some tests using the following C++ script to see what the result would be of YUV to RGB conversion:
void Test1() { int width = 1; int height = 1; unsigned char* rgb_image = new unsigned char[width * height * 3]; //width and height of the image to be converted int y; int cr; int cb; double r; double g; double b; int i = 0; //first pixel y = 21; cb = 122; cr = 139; r = y + (1.4065 * (cr - 128)); g = y - (0.3455 * (cb - 128)) - (0.7169 * (cr - 128)); b = y + (1.7790 * (cb - 128)); //This prevents colour distortions in your rgb image if (r < 0) r = 0; else if (r > 255) r = 255; if (g < 0) g = 0; else if (g > 255) g = 255; if (b < 0) b = 0; else if (b > 255) b = 255; rgb_image[i] = (unsigned char)r; rgb_image[i + 1] = (unsigned char)g; rgb_image[i + 2] = (unsigned char)b; unsigned char rr = (unsigned char)r; unsigned char gg = (unsigned char)g; unsigned char bb = (unsigned char)b; ////second pixel //y = yuyv_image[j + 2]; //cb = yuyv_image[j + 1]; //cr = yuyv_image[j + 3]; //r = y + (1.4065 * (cr - 128)); //g = y - (0.3455 * (cb - 128)) - (0.7169 * (cr - 128)); //b = y + (1.7790 * (cb - 128)); //if (r < 0) r = 0; //else if (r > 255) r = 255; //if (g < 0) g = 0; //else if (g > 255) g = 255; //if (b < 0) b = 0; //else if (b > 255) b = 255; //rgb_image[i + 3] = (unsigned char)r; //rgb_image[i + 4] = (unsigned char)g; //rgb_image[i + 5] = (unsigned char)b; }This also resulted in the same values for RGB:
Obtained from: https://stackoverflow.com/questions/9098881/convert-from-yuv-to-rgb-in-c-android-ndk
This really leads me to think that the issue should be in or before the CopyColorsTo method
Reacted by James Jackson-SouthReacted by Anton Firszov@devedse your feedback and investigation is really useful!
I'm planning to have a deep dive back into the Jpeg decoder in ~4 weeks. I will start by analyzing and TDD-ing the IDCT + color conversion chain based on your observations.
Reacted by James Jackson-South@JimBobSquarePants I did some more testing and was initially positively surprised that Image1 now returned the right pixel data for pixel X: 5990 Y: 3992.
However when I ran the 2 images through the whole compare again it failed on pixel X: 0 Y:0 directly :(.
This time pixel1 (from image1) had the correct data, pixel2 (from image2) again differed by a tiny bit.
I think there's always going to be minor differences in decoders. I've read that System.Drawing even differs slightly on 32 and 64bit machines.
I've tried to keep casting to an absolute minimum in my test decoder to reduce error.
I've changed up my library to now first use LibVIPS to convert the image from JPEG to PNG before doing the comparison. Doing that will actually give me equal images again.
So basically (pseudo code):
public void AreImagesEqual(image1, image2) { if (image1 == jpg) image1 = Vips.ConvertToPng(image1) if (image2 == jpg) image2 == Vips.ConvertToPng(image2) //Do the image comparison using ImageSharp here. }@devedse We're in the midst of a complete rewrite of the jpeg decoder which will likely utilise floating point maths throughout (so we can use SIMD). We'll revisit this once we have completed that work.
I think we've done enough work now to ensure adequate accuracy for our decoder.
Different decoders yield different results with varying levels of consistency per image. We're well within acceptable parameters.
Description
I'm writing a small tool that compares images after doing lossless optimizations on them using another tool: https://encode.ru/threads/1589-FileOptimizer
I took a picture with my own Camera (6000x4000) and ran it through the FileOptimizer. After that I used the tool I created to compare the source and target image to see if there were any pixel differences. The comparison failed on pixel X: 5990 Y: 3992
For image1 the debugger shows: 36, 15, 10, 255 (RGBA)
For image2 the debugger shows: 36, 15, 12, 255 (RGBA)
The strange this is, if I open the same images in Paint.NET the pixel value is: 38, 14, 12, 255 (RGBA). (And it's the same for both images).
I also tested what the pixel value would be of the images saved as BMP. In Paint.NET they read again as the same values we saw before:
Steps to Reproduce
Images that should be equal:
SourceImagesThatShouldBeEqual.zip
Code I'm using to reproduce the bug:
System Configuration