Reading files from within const eval #44

Activity

1. oli-obk commented on Jun 25, 2020

   

   oli-obk

   on Jun 25, 2020

   Last edited by oli-obk

   Contributor

   More actions

   by invoking the machinery that powers include_bytes!.

   It's not yet possible to share the machinery between include_bytes and the hypothetical machinery for const eval reading files, because include_bytes runs at a time where there are no queries available yet. In order for const eval to be able to read from files, we need to add a query that goes from PathBuf to &'tcx [u8] by reading the entire file in one go and interning it. This means that if you have a

   const fn foo() -> String {
       File::open("foo.txt").unwrap().read_to_string()
   }

   you will get different results when calling this function from different crates, because each crate will resolve foo.txt to a different file (to this_crate_root/foo.txt, so right next to Cargo.toml).

   I don't think we need to take care to ignore the target directory, because include_bytes can also read from that. Since we're caching all read files via the query system, we won't ever get a situation where the files differ between each read.

   Two important thing we need to take care of:

   1. Prevent any non-relative paths or relative paths that go outside the crate root. We can have different schemes to do that. One variant would be to treat the crate root as the file system root.
   2. Ensure the query is never cached in the incremental cache. Even if none of its inputs or input queries change we need to reread the entire file in order to check if the file changed.
2. RalfJung commented on Jun 25, 2020

   

   RalfJung

   on Jun 25, 2020

   Member

   More actions

   In order for const eval to be able to read from files, we need to add a query that goes from (CrateNum, PathBuf) to &'tcx [u8] by reading the entire file in one go and interning it. This means that if you have a

   That would take care of dynamically computing the filename. However, @joshtriplett also asked to be able to read parts of a file without reading the whole file, which this would not do.

   Prevent any non-relative paths or relative paths that go outside the crate root. We can have different schemes to do that. One variant would be to treat the crate root as the file system root.

   Why that? Does include_bytes! do that? Certainly build.rs and proc macros can read anywhere on the file system... (IMO they should be sandboxed but that's a long and separate discussion.^^)

   Two important thing we need to take care of:

   3. Ensure that the query system never recomputes this query. Currently, for many queries dropping the cache is okay because the queries are all pure functions that can be recomputed any time. Reading from a file might be the first query to not have that property, so the entire query system needs to cooperate (and future changes in the query system need to take this into account).
3. oli-obk commented on Jun 25, 2020

   

   oli-obk

   on Jun 25, 2020

   Contributor

   More actions

   Why that? Does include_bytes! do that? Certainly build.rs and proc macros can read anywhere on the file system... (IMO they should be sandboxed but that's a long and separate discussion.^^)

   Because... we can do it? No need to start out a new feature without sandboxing if we can sandbox it from the start

   Ensure that the query system never recomputes this query. Currently, for many queries dropping the cache is okay because the queries are all pure functions that can be recomputed any time. Reading from a file might be the first query to not have that property, so the entire query system needs to cooperate (and future changes in the query system need to take this into account).

   That's actually fine, because if recomputing the query causes the result to differ, it will poison all queries that called this query and force them to get recomputed, too. So you don't get any behaviour where if you have a dataflow from such a file to an array length, the array length changes within the same compilation.
4. joshtriplett commented on Jun 27, 2020

   

   joshtriplett

   on Jun 27, 2020 via email

   MemberAuthor

   More actions

   On Wed, Jun 24, 2020 at 11:57:26PM -0700, Ralf Jung wrote: > In order for const eval to be able to read from files, we need to add a query that goes from (CrateNum, PathBuf) to &'tcx [u8] by reading the entire file in one go and interning it. This means that if you have a That would take care of dynamically computing the filename. However, @joshtriplett also asked to be able to read parts of a file without reading the whole file, which this would not do.

   That would be nice, but not necessary right away. I'd like the `File` API to just work in const eval, but it's fine for now if it just reads the entire file into memory and doesn't try to read only a subset.
5. septatrix commented on Aug 12, 2023

   

   septatrix

   on Aug 12, 2023

   More actions

   Calling a const function at compile-time will always yield the same result as calling it at runtime, even when called multiple times.
   ~ https://doc.rust-lang.org/reference/const_eval.html#const-functions

   How could this still hold with arbitrary file access which whose content can change at any point?
6. ayebear commented on May 5, 2024

   

   ayebear

   on May 5, 2024

   Last edited by ayebear

   More actions

   How could this still hold with arbitrary file access which whose content can change at any point?

   Why does const fn have to be deterministic if proc macros don't have to be? Couldn't we just accept that results could be different at compile time because of files/floats, and then consistently apply that to all of the compile-time concepts? Or just make these rules warnings instead of errors.
7. RalfJung commented on May 5, 2024

   

   RalfJung

   on May 5, 2024

   Last edited by RalfJung

   Member

   More actions

   Proc macros are like code that generates .rs files. They are executed once upfront and then produce the input for the rest of complication.

   Constants can be evaluated multiple times. E.g. when you have something like

   impl<T, U> Trait for Type<T, U> {
     const C: Type = ...;
   }

   The value of C can only be computed once T and U are known. But when two different crates independently compute C for the same T and U, it is crucial that they get the same result.