Read IOStream for use with S3 #42

Activity

1. dahong67 commented on Jun 14, 2024

   

   dahong67

   on Jun 14, 2024

   Last edited by dahong67

   More actions

   Would be great for this to get a more "proper" fix, but I think a simple (inefficient?) work-around for now could be something along the lines of:

   function matread(io::IO)
       # Write to a temporary file
       tempfile = tempname()
       write(tempfile, io)
   
       # Read data from temporary file
       data = matread(tempfile)
   
       # Clean up temporary file and return data
       rm(tempfile)
       return data
   end

   Would need a bit of work to make sure errors, etc. are handled correctly (e.g., what should be done if io is empty?).

   @JeffFessler, is this something you're still looking at?
2. JeffFessler commented on Jun 14, 2024

   

   JeffFessler

   on Jun 14, 2024

   Member

   More actions

   I found a workaround of the form data = matread(Downloads.download(dataurl)) for my own use, e.g.,
   https://juliaimagerecon.github.io/Examples/generated/mri/5-l-plus-s/#Read-data
   (You might be amused to see whose work is cited there.)

   That does not address the OP question though, and it still (inefficiently?) uses an intermediate file.

   ❤️React with ❤️1Reacted by David Hong
3. dahong67 commented on Jun 17, 2024

   

   dahong67

   on Jun 17, 2024

   More actions

   You might be amused to see whose work is cited there

   Fun! Just showed her. :)
4. matthijscox commented on Apr 7, 2026

   

   matthijscox

   on Apr 7, 2026

   Last edited by matthijscox

   Member

   More actions

   This should be doable for the old file formats (before v7.3), which are just binary files.

   The new format (v7.3) is HDF5. I dont think HDF5.jl supports IOBuffer input. Reading the HDF5.API source code, you can see it calls into the libhdf5 C library. We currently use h5f_open inside MAT.jl, which uses a filepath. I'm not seeing a streaming alternative there. I also can't find information on a libhdf5 streaming options on the internet anywhere yet.

   in-memory file support does seem possible: HDF5.jl In-memory-HDF5-files, so maybe we can use that? Do you really need streaming? Or can we just read the full IOBuffer into a byte vector, then turn it into an in-memory HDF5 object, and then read that with MAT.jl? Note this would also solve issue #96