Repository navigation
remove language-level UB for non-UTF-8 str #71033
Description
Activity
Would this mean that the following stops being UB?
use std::mem; fn main() { // Being valid utf8 is still a safety invariant of `str`. // As any method using `str` may depend on this invariant, // it would still be UB to use `str::from_utf8_unchecked` // or `str::as_bytes` here. let s: &str = unsafe { std::mem::transmute(b"\xff\xff" as &[u8]) }; let bytes: &[u8] = unsafe { std::mem::transmute(s) }; assert_eq!(bytes, &[0xff, 0xff][..]); }
@lcnr yes exactly.
it would still be UB to use
str::from_utf8_unchecked
orstr::as_byteshere.To be more precise, it would be library UB -- the way the methods are implemented right now, there is actually no language-level UB and nothing that Miri could possibly find, but the library is permitted to change in the future in ways that would make this language UB.
Reacted by lcnr, J. Frimmel, Aaron Hill, Elichai Turkel, Tux3, matthieu-m, Chris Wong and Ashish MylesDear community and language team.
As Ralf notes, there are no compelling reasons to keep this Undefined Behavior (UB) at the level of the abstract machine in terms of the validity invariant of
str. Therefore, keeping it UB at this level only complicates the language definition instead with no notable benefits.Instead, we can make this a library invariant, and leave it as "library UB" or "unspecified behavior". Indeed, this is probably what we always meant by the note in the reference. I hereby propose that we accept that new definition:
@rfcbot merge
Team member @Centril has proposed to merge this. The next step is review by the rest of the tagged team members:
No concerns currently listed.
Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!
See this document for info about what commands tagged team members can give me.
there is actually no language-level UB and nothing that Miri could possibly find
This leads to an interesting question: what can miri find?
My first guess would be someunsafe { unreachable_unchecked() }call in UTF-8 decoding.It would be great to figure out if there is something like this that miri does detect, even if miri stops checking
strvalues for UTF-8 validity altogether, and use it as a test case.Reacted by jyn and AmadeusineIt would be great to figure out if there is something like this that miri does detect, even if miri stops checking str values for UTF-8 validity altogether, and use it as a test case.
Note that Miri does not check behind references, so while
strwould be checked, that type is basically unused, and&stris not checked.This leads to an interesting question: what can miri find?
My first guess would be some unsafe { unreachable_unchecked() } call in UTF-8 decoding.I scrolled over
str/mod.rsto see how the invariant gets used. I certainly missed some things, but here is what stood out:- Calling
char::from_u32_unchecked, which must be a valid unicode codepoint. Miri checks this. - The searching/splitting talks a lot about indices being at unicode boundaries. I do not know what happens if they are not.
- I suspect somewhere it might also lead to out-of-bounds accesses when it thinks there is a 4-byte character following, but only 2 bytes are left in the buffer.
- I did not find any
unreachable/unreachable_unchecked.
Reacted by Eduard-Mihai Burtescu- Calling
This seems entirely reasonable to me. If you never call any of
str's functions, just storing non-UTF-8 in it shouldn't cause any issue.I wonder if we might be able, in the future, to carefully exclude a few of
str's functions from the "library UB" requirements.cc @BurntSushi: Would this change potentially simplify
bstr?I wonder if we might be able, in the future, to carefully exclude a few of str's functions from the "library UB" requirements.
Yes I think that is definitely possible. It is somewhat similar to, for example, how we promise that
Vec::pushwill keep existing pointers working if it does not reallocate (though that is AFAIK not very clearly documented): library methods can make extra promises beyond the ways they could be used in safe code.27 remaining items
A minor point regarding terminology: Aren't these "UTF8 encoding units" called (UTF-8) code units in established Unicode parlance? Which, incidentally, is probably an argument in favor of adding a named type as it would simply reify an existing concept rather than introducing an ad-hoc one!
EDIT: Skip my post and read the next one
@jdahlstrom Good point: http://www.unicode.org/glossary/#code_unitUtf8CodeUnitwould be a much better name.Yes, “code unit” is established in Unicode but the code unit for UTF-8 is
u8, not a specialization ofu8that has validity invariants of its own. It’s only for a UTF-8 sequence of bytes (a.k.a. 8-bit code units) that Unicode defines being “well-formed”.The formal definitions for this start at https://www.unicode.org/versions/Unicode13.0.0/ch03.pdf#G7404
FCP was originally requested for changing the validity invariant of
strto that of[u8], but during discussion consensus seems to have shifted towards rather using that of[Utf8Byte]with#[rustc_layout_scalar_valid_range_end(0xf7)] struct Utf8Byte(u8);
(the name of that type is still up for bikeshedding)
I am a bit confused now about what this FCP is actually deciding, if/when it completes.
Mmm, then maybe it makes sense to also remove language-level UB for non Unicode scalar
chars too?Mmm, then maybe it makes sense to also remove language-level UB for non Unicode scalar
chars too?Rust does exploit the limited values that can be stored in a
char:Here, returning
NoneforOption<char>is the same as returning0x110000u32which is one past the largest unicode scalar value.
https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=ad1776519737e67fbe35bf76bb8451c4@RalfJung I think there’s consensus that such a definition for
Utf8Bytewould not be wrong or harmful. It’s much less clear (at least to me) that it’s actually useful. What actual optimization would thisrustc_layout_scalar_valid_range_end(potentially) enable, given thatstronly ever exists behind a pointer indirection such as&str?Process-wise, my understanding is that an FCP finishing means accepting a proposal as it was when that FCP was proposed. If new information or consensus emerges later (especially if it’s after some votes) and a team member feels the original proposal should not be accepted, they should file a concern or cancel the FCP. (And potentially propose a new FCP for a different proposal.)
Reacted by Martin Habovštiak and KornelIf there's no potential optimizations to apply it seems better to completely remove the validity invariant as proposed, to allow temporarily using the backing buffer from
str::as_bytes_mutwith less manually checked safety. With a non-compiler-checked validity invariant on those bytes it makes working with an&mut strmuch more error-prone.Alternatively, exposing
Utf8Byte, anunsafe fn as_utf8_bytes_mut(&mut self) -> &mut [Utf8Byte], some checked operations forUtf8Byte, and some way to wrap an arbitrary&mut [u8]into an&mut [Utf8Byte](I'm not sure if pre-validating then transmuting this would be valid even withrepr(transparent)with the additional layout constraints); all together would allow building useful APIs on&mut strwith lessunsafecode.I agree with @SimonSapin. I would not bother with the "encoding units" myself, which seems like an extra layer of complexity that doesn't add much of practical value.
Reacted by Eduard-Mihai BurtescuThe final comment period, with a disposition to merge, as per the review above, is now complete.
As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.
The RFC will be merged soon.
So the FCP that passed means the decision was that
stris like[u8]?Reacted by Jacob Lifshay and AmadeusineI believe so.
This is the Rust-side issue for rust-lang/reference#792 just so that we can use fcpbot. The change description follows.
Ever since Rust 1.0, the reference said that a non-UTF-8
strcauses immediate UB. In terms of today's terminology, that means thatstrhas a validity invariant of being valid UTF-8.However, that seems unnecessary: the compiler does not actually exploit this, nor is there any clear way it could exploit this. Making UTF-8 a library-level safety invariant is more than enough for everything
strdoes. Most likely, it was made a validity invariant because we had not yet properly teased apart those two concepts when the document was initially written.This is also the conclusion that the UCG WG arrived at in rust-lang/unsafe-code-guidelines#78.
I therefore propose we remove the UTF-8 clause from the language spec, so that
strwill have the same validity invariant as[u8].