Repository navigation
to_ascii_uppercase and to_ascii_lowercase operate on non-ASCII characters #31203
Description
Behold!
#[test]
fn liar_liar_pants_on_fire() {
use std::ascii::AsciiExt;
assert_eq!("café".to_ascii_uppercase(), "CAFÉ");
assert_eq!("café".to_ascii_uppercase(), "CAFé");
assert_eq!("CAFÉ".to_ascii_lowercase(), "café");
assert_eq!("CAFÉ".to_ascii_lowercase(), "cafÉ");
}This is obviously silly. The problem is that this is running an ASCII-only operation on Unicode strings without actually dealing with their Unicode-ness.
These functions should either correctly deal with grapheme clusters (by ignoring them since they're not in ASCII), or document that it does not correctly handle grapheme clusters, preferably with an example (like the above).
(Actually having a standard Ascii type would be even more muchly preferable, but I suspect that's way out of scope.)
Activity
steveklabnik commentedon Jan 26, 2016
This is the documented behavior: http://doc.rust-lang.org/std/ascii/trait.AsciiExt.html#tymethod.to_ascii_uppercase
/cc @rust-lang/libs
frewsxcv commentedon Jan 26, 2016
In case anyone is curious why this works:
caféis actually representedcafe\u{301}https://en.wikipedia.org/wiki/Combining_character
tbu- commentedon Jan 27, 2016
@steveklabnik It would be nice if the documentation could mention that it operates on codepoints, not grapheme clusters.
frewsxcv commentedon Jan 27, 2016
Also, worth considering adding one of the examples above to go with it
steveklabnik commentedon Jan 27, 2016
@tbu- nothing in the standard library operates on grapheme clusters, though. I'm not sure what the right level of repeating this is.
Maybe @frewsxcv is right here, and an example is the best way for this particular function.
tbu- commentedon Jan 27, 2016
I think it's confusing that it talks about characters when it exhibits the weird behaviour mentioned above. Maybe it could just explicitly codepoints?
steveklabnik commentedon Jan 27, 2016
charis a unicode scalar value. So 'character' is always referring to a codepoint, not a grapheme.tbu- commentedon Jan 27, 2016
Yes, it's confusing naming unfortunately. Maybe we could add these unexpected test cases to the documentation?
DanielKeep commentedon Jan 28, 2016
For what it's worth, although I'd prefer this method do "the right thing", simply having an example to make the behaviour with respect to combining code points clear would be a reasonable solution.
bluss commentedon Jan 28, 2016
@steveklabnik Note that this function may transform "é" to "É", which is the "ascii violating" behavior
It needs pointing out in the doc. However, we don't need to mention grapheme clusters. It is application dependent which algorithms you use over your unicode data. Implying that grapheme clusters is always the right thing to do is misleading. Rust strings are brilliant as they are in providing minimal unicode consistency with low overhead.
steveklabnik commentedon Jan 28, 2016
Yes, I would also think that examples of this are a good idea.
frewsxcv commentedon Feb 4, 2016
I took a stab at this: #31401
Rollup merge of rust-lang#31401 - frewsxcv:clarify-ascii, r=steveklabnik