Repository navigation
Regex is incorrectly handling casing of some ranges #36149
Description
using System;
using System.Text.RegularExpressions;
class Program
{
static void Main()
{
Console.WriteLine(Regex.IsMatch("\xF7", @"^(?i:[\xD7\xD8])$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant));
Console.WriteLine(Regex.IsMatch("\xF7", @"^(?i:[\xD7-\xD8])$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant));
}
}The above two patterns should be identical, with on character set containing \xD7 and \xD8 and the other containing the range from \xD7 through \xD8 (which is just \xD7 and \xD8, since there's nothing in between them).
However, the first correctly prints false whereas the second incorrectly prints true.
The implementation handles casing by creating a character class that's the lowercased version of the original. That means that for individual characters, it just adds the lowercase character:
and for ranges it needs to add the lowercase character for each character in the range:
In the first case above, it follows the first path, adding in the ToLower(\xD7) (which is just \xD7) and the ToLower(\xD8) (which is \xF8).
In the second case, however, it follows the second path, and ends up incorrectly adding a range from \xF7 through \xF8.
As a result, the second case ends up incorrectly matching \xF7.
Activity
pgovind commentedon Sep 14, 2020
Just adding some notes here: The range
/xD7 - /xD8is215 - 216in decimal.char.ToLower(/xD7)is/xD7i.e.215.char.ToLower(/xD8)is however248. Internally, we use as_lcTableto map capital letters to small letters. However, the mapping logic inAddLowercaseRangeassumes that the capital lettersrange.Firstandrange.Lastcan be converted torange.First+32andrange.Last+32respectively. Therefore, we end up adding the range247 - 248in theAddLowercaseRangemethod! (which is why we match/xF7(which has the decimal value247).I have some questions (@GrabYourPitchforks or @tarekgh perhaps?) before I can put up a fix:
/xD7(decimal value of 215) does NOT appear in the list. Let's call this a "hole". Does this mean/xD7is special somehow? Are there other such holes in the unicode spec? If so, all those cases will probably be bugs in our current implementation./xD7has a special Unicode category, that would make it easy to put up a fix. If not, is there an easy way to determine how many "holes" exist given a start and end point?A fix such as checking if (
char.ToLower(something) == char.ToUpper(something)) might work, but I'd like to get more information before I test it.GrabYourPitchforks commentedon Sep 15, 2020
U+00D7is a symbol (see https://www.fileformat.info/info/unicode/char/00d7/index.htm), so it doesn't have an associated case.You'll occasionally see symbols and other non-alpha characters sitting in the middle of character blocks. You'll also see some ranges where the uppercase and lowercase variants aren't separated by 32. For example, Ģ (
U+0122) is the uppercase form of ģ (U+0123). That's a separation of only 1 code point, not 32 code points.Best thing to do might be to case-map each code point independently, then see if you can generate any ranges from those maps. For example, consider that you're given the regex "allow 100 - 109, case-insensitive." And the lowercase mappings are as follows:
Then ideally we'd add the ranges
[100 - 109],[132 - 134], and[135 - 140](all inclusive) to the match list. With some additional logic we could compact overlapping or adjacent ranges.Also worth pointing out:
string.Contains(a, b, StringComparison.OrdinalIgnoreCase)normalizes to uppercase before performing the check. HasRegex.IsMatch(..., IgnoreCase)always normalized to lowercase? This could result instring.ContainsandRegex.IsMatchreturning different results for the same inputs. It might be worth investigating getting these two implementations in sync so as not to surprise callers.GrabYourPitchforks commentedon Sep 15, 2020
BTW, here are the two resources I use for getting information on Unicode code points.
https://unicode.org/cldr/utility/character.jsp is the authoritative source. It also has click-through navigation, so you can ask for things like "show me all code points which, when uppercased, convert to the code point I'm looking at right now." To use it, either paste the character itself into the text box at the top of the page, or write it as a hex-formatted number (padded to at least 4 digits, no 0x prefix). Example:
00D7or12345.https://www.fileformat.info/info/unicode/char/0000/index.htm is also a useful resource. It will show you the UTF-8 or UTF-16 encodings of the scalar value, and it will also show you the best attempt at rendering the glyph, even if you don't have an appropriate font pack installed locally. To use it, replace the "0000" in the URL above with the hex-formatted number for your code point (padded to at least 4 digits, no 0x prefix). Example: /00D7/index.htm or /12345/index.htm.
stephentoub commentedon Sep 15, 2020
Yup
pgovind commentedon Sep 21, 2021
#42282 is being reverted in #59425. Re-opening this issue.
joperezr commentedon Apr 5, 2022
I have validated that #67184 fixes this