Repository navigation
dump: add caching for additional speed up #12
Description
The dump commands reads all blobs directly from the backend without caching them.
So the idea is to add a cache, and use the fact that we know a priori which blobs we need and in which order:
- ideally (if we have enough memory) save all later needed blobs in the cache.
- when a cached blob is not used in future, we can immediately remove it from the cache to free memory as early as possible.
- add a '--cache-size` size option. We can calculate a priori which blobs must be removed from cache to get an optimal cache.
Activity
aawsome commentedon Jul 8, 2023
Another idea would be to parallelize the reading of blobs. As we already know the list and order of blobs to read, this can be trivially parallelized using e.g.
pariter.aawsome commentedon Jul 8, 2023
A third idea (but more involved): Not read blob-by-blob, but identify if contiguous parts of a packfile can be used. If so, read the whole part and process the blobs as-needed.
[-]speed up dump command[/-][+]`dump`: add caching for additional speed up [/+]perf: parallelize blob fetch in dump command (rustic-rs#501)