Repository navigation
Writing valid UTF-8 to the Windows terminal using stdout().write() can incorrectly return an error #83258
Description
I tried this code:
use std::io::Write;
fn standard_write_all<W: std::io::Write>(mut w: W, mut buf: &[u8]) -> Result<(), std::io::Error> {
while !buf.is_empty() {
match w.write(buf) {
Ok(0) => {
return Err(std::io::Error::new(
std::io::ErrorKind::WriteZero,
"failed to write whole buffer",
));
}
Ok(n) => buf = &buf[n..],
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {}
Err(e) => return Err(e),
}
}
Ok(())
}
fn main() {
let poison = "Text here does not matter.\n".to_owned() + "x".repeat(1024 - 1).as_str() + "😀";
// The special stdout().write_all() impl works.
std::io::stdout().write_all(poison.as_bytes()).unwrap();
println!();
println!();
// A standard write_all() implementation fails.
//
// 1. First write call
// The bytes up to the newline are passed through to the console in one write.
//
// The next 1023 b'x' bytes and the first byte of "😀" are buffered leading to
// an incomplete codepoint at the end of the buffer. The buffer is now poisoned
// meaning that the next write() call will fail even if the missing UTF-8 bytes
// are supplied.
//
// 2. Second write call
// The buffer is flushed completely on the next call, the console writes 1023 b'x' bytes
// and is then forced to write the remaining byte from the incomplete UTF-8 sequence.
standard_write_all(std::io::stdout(), poison.as_bytes()).unwrap();
}I expected to see this happen: Program does not panic.
Instead, this happened: Program panics with the error Windows stdio in console mode does not support writing non-UTF-8 byte sequences.
Meta
rustc --version --verbose:
rustc 1.50.0 (cb75ad5db 2021-02-10)
binary: rustc
commit-hash: cb75ad5db02783e8b0222fee363c5f63f7e2cf5b
commit-date: 2021-02-10
host: x86_64-pc-windows-msvc
release: 1.50.0
Backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidData, error: "Windows stdio in console mode does not support writing non-UTF-8 byte sequences" }', src\main.rs:35:65
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\std\src\panicking.rs:493
1: core::panicking::panic_fmt
at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\panicking.rs:92
2: core::option::expect_none_failed
at /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b\/library\core\src\option.rs:1268
3: core::result::Result<tuple<>, std::io::error::Error>::unwrap<tuple<>,std::io::error::Error>
at C:\Users\hans\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\result.rs:973
4: termcolorbug::main
at .\src\main.rs:35
5: core::ops::function::FnOnce::call_once<fn(),tuple<>>
at C:\Users\hans\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Update
To clarify: This problem occurs with all Write trait implementations that don't override the default Write::write_all() implementation, e.g. with this simple DelegatingWrite:
use std::io::Write;
struct DelegatingWrite<W> {
delegate: W,
}
impl<W: Write> DelegatingWrite<W> {
fn new(delegate: W) -> DelegatingWrite<W> {
DelegatingWrite { delegate }
}
}
impl<W: Write> Write for DelegatingWrite<W> {
fn write(&mut self, b: &[u8]) -> std::io::Result<usize> {
self.delegate.write(b)
}
fn flush(&mut self) -> std::io::Result<()> {
self.delegate.flush()
}
}
fn main() {
let poison = "Text here does not matter.\n".to_owned() + "x".repeat(1024 - 1).as_str() + "😀";
let mut w = DelegatingWrite::new(std::io::stdout());
write!(w, "{}", poison).unwrap(); // <-- panics
}
Activity
Count-Count commentedon Mar 18, 2021
I see two distinct possibilites to resolve this:
Treat incomplete
write()s not ending on a codepoint boundary as a bug and fix that in the stdlib. Also clarify in the docs that indivualwrite()calls must be passed complete UTF-8 sequences if a Windows terminal is attached.In the vain of "Be liberal in what you accept" handle incomplete UTF-8 passed in via
write()calls appropriately, as long as the sequence is completed properly in the nextwrite()call(s).ChrisDenton commentedon Mar 18, 2021
Hm, so this is ultimately a bug in the buffer
std::io::stdout()uses? It attempts to write 1024 bytes but only 1023 bytes can written in the first OS call. It then has a leftover byte, which I guess it tries to write by itself thus causing this error?Count-Count commentedon Mar 18, 2021
@ChrisDenton The logic which errors if incomplete UTF-8 is passed in at index 0 is in the std lib at sys/windows/stdio.rs. That is where the error message comes from.
As far as I can see the stdio buffer implementation (in stable, it is in linewritershim.rs, apparently currently being revamped) can not know of the requirement that only complete UTF-8 may be written to the underlying Windows terminal so I think that it should be encapsulated in sys/windows/stdio.rs by just accepting and buffering incomplete UTF-8 when necessary.
ChrisDenton commentedon Mar 18, 2021
I mean, a generic buffer can attempt to avoid small writes (say, less than 16 bytes) by copying those left over bytes to the front of the buffer and waiting for either another
writeto refill the buffer or aflushto empty it.But I see what you mean. Introducing a small buffer to windows/stdio would fix the problem.
[-]Writing valid UTF-8 to the Windows terminal using stdout().write() can panic.[/-][+]Writing valid UTF-8 to the Windows terminal using stdout().write() can incorrectly return an error[/+]Rollup merge of rust-lang#83342 - Count-Count:win-console-incomplete-…
Auto merge of rust-lang#83342 - Count-Count:win-console-incomplete-ut…
Auto merge of rust-lang#83342 - Count-Count:win-console-incomplete-ut…