Repository navigation
Computed constants output misrepresents source locations if non-ASCII characters are present in the source #22
Description
If the script that jsir_gen is acting upon contains non-ASCII characters such as Cyrillic script (even in comments), it mixes up byte-position vs character-position and outputs wrong segments of the script. The issue appears to be that JsPosition when fetching from the JSON returns character position, but then original_source.substr on lines 75-77 of jsir_gen_lib.cc uses absl::string_view.substr which operates on bytes.
For example, if you run jsir_gen on the script:
// тестовые символы кириллицы
let c = atob("SGVsbG8sIHdvcmxkIQ==");It will output:
// Dataflow etc omitted
================================================================
Computed constants:
================================================================
From [39, 67): `риллицы
let c = atob` -> `Hello, world!`
================================================================
In between the ` ` it has mistakenly captured the newlines and the end of the comment due to the mixup instead of the actual function call.
Activity
enbyte commentedon Aug 15, 2025
After reading about this a little it appears the two possible solutions are to use a heavy-duty i18n library like ICU, which seems like a massive waste for one function, or to use a hacky homebrew iterator that manually implements getting a charpos and going through the bytes. Both seem unpleasant, and I'm not sure if the right path here is correct and wasteful or fast and hacky. What are your thoughts? If a custom function seems like the better choice I am happy to make a PR implementing it.
phisiart commentedon Aug 16, 2025
Thanks for reporting this bug!
I did an experiment on astexplorer with the following code:
It seems that source locations are based on utf16 code points:
"a"contains 3 ASCII characters == 3 utf16 code points;"и"contains one non-ASCII characterи, but this character can be represented using one utf16 code point;"😀"contains one non-ASCII character😀which requires 2 utf16 code points.Currently, all our strings are utf8, so at the very least, we need to convert it to
std::u16string(maybe using https://github.com/nemtrif/utfcpp) before takingsubstrs. I don't know if it's worthwhile to just consistently usestd::u16stringthroughout the codebase though.