Repository navigation
Do tests really use images from 'ReferenceOutput' folder? #2158
Description
Activity
Nice catch! Looks like a bug sneaked into our
PngEncoderTestscode and we are comparing the image to itself instead of the reference file :)ImageSharp/tests/ImageSharp.Tests/Formats/Png/PngEncoderTests.cs
Lines 624 to 628 in 75dc96a
Other encoder tests seem to be correct.
Reacted by James Jackson-SouthI was writing tests for jpeg encoder PR and looked at the tests which compare saved images to reference images
Note that the mentioned PNG tests are validating generated images. Tests which are working with file inputs are using a utility method
VerifyEncoder:ImageSharp/tests/ImageSharp.Tests/Formats/Bmp/BmpEncoderTests.cs
Lines 364 to 365 in 2aa150c
It compares the original image to the one re-decoded from the ImageSharp encoder output by a reference decoder, therefore it doesn't need reference images. Questionable approach for Jpeg, but still used in the current tests.
Maybe we should create a set of tests with smaller images and create optimized reference PNG-s instead? This would raise the question if we should use ImageSharp or some other software for generating the reference output.Scratch that doesn't make sense, I don't have a better idea for result verification than the re-decoding trick.Questionable approach for Jpeg, but still used in the current tests.
I don't really have an idea how to compare new output color types in jpeg encoder. Funny thing, current tests found a bug in my PR :D
@antonfirsov We actually do the same in the Bmp Encoder tests for quantized images.
What would be a solution to fix this issue?
- Use
VerifyEncoderinTestPngEncoderCoreand delete the PngEncoder reference images or - Assume the reference images are correct and use
CompareToReferenceOutput()instead?
- Use
I'll have to re-read our tests. As a side I really want to put some time to focus on cleaning up our test suite. It's difficult to navigate.
Prerequisites
DEBUGandRELEASEmodeImageSharp version
main branch
Other ImageSharp packages and versions
None
Environment (Operating system, version and so on)
Windows 10
.NET Framework version
net6
Description
I was writing tests for jpeg encoder PR and looked at the tests which compare saved images to reference images from 'ReferenceOutput' folder. Thing is, I don't see anything related to that folder in test code. Just to be sure I've deleted 'ReferenceOutput/PngEncoderTests' folder and all of png encoder tests still pass.
Most likely I've missed something but I just can't see any connections to this folder in test code.
Steps to Reproduce
Delete 'ReferenceOutput/PngEncoderTests' folder and launch png tests.
Images
No response