Remove two redundant expressions in scalar UTF8 validation

[BenjaminBucher]

During experimentation, I was playing around with the scalar utf8 validation code and identified two expressions that by necessity always evaluate to the same value (false).

simdutf/include/simdutf/scalar/utf8.h

Lines 56 to 60 in 427a6f5

	// range check
	code_point = (byte & 0b00011111) << 6 | (data[pos + 1] & 0b00111111);
	if ((code_point < 0x80) || (0x7ff < code_point)) {
	return false;
	}

code_point = (byte & 0b00011111) << 6 | (data[pos + 1] & 0b00111111);
if ((code_point < 0x80) || (0x7ff < code_point)) {
    return false;
}

• code_point is a uint32_t
• byte is a unsigned char
• data is a unsigned char* (Note: this might not be entirely true. It's definitely something comparable, but I don't believe it makes a difference)

The way code_point is evaluated, only the 11 least significant bits can be active.
0x7ff is the number with exactly the 11 least significant bits active.
So the expression 0x7ff < code_point always evaluates to false, as at least the twelfth bit would need to be active which isn't possible.

The same goes for the later check

simdutf/include/simdutf/scalar/utf8.h

Lines 72 to 79 in 427a6f5

	// range check
	code_point = (byte & 0b00001111) << 12 |
	(data[pos + 1] & 0b00111111) << 6 |
	(data[pos + 2] & 0b00111111);
	if ((code_point < 0x800) || (0xffff < code_point) ||
	(0xd7ff < code_point && code_point < 0xe000)) {
	return false;
	}

with the expression 0xffff < code_point.

Here is the relevant code in the compiler explorer: https://godbolt.org/z/9r4fhqW7E.
Note that removing either check reduces the amount of instructions produced, meaning the compiler did not catch this optimization (on -O3).

That also made me skeptical, but the compiler indeed optimizes those checks out in a simpler example: https://godbolt.org/z/M135ejWea.

These two checks each appear twice in the following files:

include/simdutf/scalar/utf8_to_utf16/utf8_to_utf16.h
include/simdutf/scalar/utf8_to_utf32/utf8_to_utf32.h
include/simdutf/scalar/utf8.h

Also in tests/reference/validate_utf8.cpp, but as a reference implementation, it should probably be left as-is.

These two checks also exists in ports of this library, e.g. https://github.com/simdutf/SimdUnicode/blob/2adc753b41cdc167836dc64c2af133368de19b1b/src/UTF8.cs#L125-L130.

If I didn't miss anything, I'll happily follow up with a pull request :)

[clausecker]

Nice find! I wonder what the intended logic was.

[lemire]

Please issue a PR (this way, you get credit).