borgception

[ThomasWaldmann]

i thought a bit about how to optimize the chunks cache and just wanted to document one weird idea.

the issue with the chunks cache is that it needs to match the overall repository state (== have up-to-date information about all chunks in all archives, including refcount, size, csize). when backing up multiple machines into same repo, creating an archive of one machine invalidates all chunk caches on the other machines and they need to resync their chunks cache with the repo, which is expensive.

so, there is the idea to store the chunk index into the repo also, so all out-of-sync clients can just fetch the index from the repo.

But:

• index can be large (way larger than segment size)
• when using raw hashtable it could have up to 75% unused bucket space
• index has additional information about chunks, so we should not store it unencrypted if the repo is encrypted
• the index should match the chunks in the repo, so it should not create own chunks in the repo.

So we need:

• chunking of the index into smaller pieces
• compression (unused bucket space is mostly binary zeros AFAIK)
• encryption

This pretty much sounds like we should just backup the index of repo A into a related, but separate borg repository A'. :-)

---

💰 there is a bounty for this

[level323]

I don't think it's a weird idea at all.

I was thinking along similar lines, actually.

I was initially "what iffing" the idea of storing the chunks cache inside
the main repo, but then saw benefits in storing it in a special dedicated
repo - because this opens the possibility of exploiting the known structure
of the chunks cache to minimise the size of its repo (and minimise transfer
time over slow links).

I think similar fresh approaches can be used to optimise the size of other
objects such as the archive metadata and the files cache to optimise
synchronisation of this data between machines in a
multi-machine-backing-up-to-central-repo use case.

On 8 December 2015 at 05:58, TW notifications@github.com wrote:

i thought a bit about how to optimize the chunks cache and just wanted to
document one weird idea.

the issue with the chunks cache is that it needs to match the overall
repository state (== have up-to-date information about all chunks in all
archives, including refcount, size, csize). when backing up multiple
machines into same repo, creating an archive of one machine invalidates all
chunk caches on the other machines and they need to resync their chunks
cache with the repo, which is expensive.

so, there is the idea to store the chunk index into the repo also, so all
out-of-sync clients can just fetch the index from the repo.

But:

• index can be large (way larger than segment size)
• when using raw hashtable it could have up to 75% unused bucket space
• index has additional information about chunks, so we should not
  store it unencrypted if the repo is encrypted

So we need:

• chunking of the index into smaller pieces
• compression (unused bucket space is mostly binary zeros AFAIK)
• encryption

This pretty much sounds like we should just backup the index into a
related borg repository. :-)

—
Reply to this email directly or view it on GitHub
#474.

[RonnyPfannschmidt]

how about having per segment index chunks - then each segment would have encrypted index update instructions, and those could be removed/regenerated as well

[RonnyPfannschmidt]

a yet different method could be to just have 2 files per segment, one encrypted metadata, one for the blob data it adds in order

the whole chunk index can then be reconstructed from the metadata

[ThomasWaldmann]

@level323 the files index does not need to be shared. it just remembers the mtime/size/inode/chunks info for all files of last backup, so it can quickly skip the files next time.

@RonnyPfannschmidt what's the point of a per-segment index if you have to merge 100.000 of them to get the full index?

[RonnyPfannschmidt]

@ThomasWaldmann you only need to deal with tens of thousands on a full index regen

the multi machine use case would be much less intensive on computation, since one only has to obtain increments

[ThomasWaldmann]

@RonnyPfannschmidt well, a segment has 5MB, so a 500GB repo has 100.000 segments. That's just an example, but a quite realistic one. Of course, it can be more or less depending on how much data you have.

But I still don't see how your suggestion should be efficient. Just for comparison: the normal, uncached and quite slow repo rebuild goes through ALL archives, ALL files and uses the file item chunk list stored in metadata as increment. The items metadata are stored clustered together in a few segments [not together with file content data]).

BTW, an incremental "just add on top what we already have" approach for the chunks cache only works as long as nothing is removed. Because if something is removed, also the information about it is gone, so we can't subtract. (see the PRs)

[RonnyPfannschmidt]

if each segment also tracks the removals, then the chunks index willl match the current state after applying a segment, the main problem would be to correct the reference segment of the current state on a vacuum - which is quite hard, since segments will be combined on vacuum

[pszxzsd]

No idea if it's applicable to the problem, but ZeroDB (docs), an end-to-end encrypted database, recently went open source as a Python implementation.

[RonnyPfannschmidt]

its highly unrelated and a completely different kind of data-store not suitable for borg's needs

[callegar]

Out of curiosity and with no intention whatsoever to be pressing, would be great to hear if there is progress on this idea.

If I understand correctly, the advantage of the cache is when multiple hosts are backed up to the same repo (which would help when a lot of overlap is expected). I get the impression that when a single host is backed up to a repo one can work without a cache at all without performance penalty using the rm -rf chunks.archive.d ; touch chunks.archive.d trick. Is this correct?

At the same time I now see in the FAQs that backing up multiple hosts to the same server is discouraged (at https://borgbackup.readthedocs.io/en/stable/faq.html#can-i-backup-from-multiple-servers-into-a-single-repository).

From this premises, I would like to ask if the borgception is still being pursued as a way to make the multiple hosts to one repo model workable, or is being dropped as unneeded as the single host to single repo model would not get any real benefit from it.

[FabioPedretti]

I think this was somewhat addressed with BORG_FILES_CACHE_SUFFIX, see #5433 and #5602.

[ThomasWaldmann]

@callegar I currently have no near-future plans to implement this.

Priority now is on getting 1.2 out (b4 released, rc1 next), then fixing any potential issues with that.

After that: working on helium milestone (which is crypto improvements). Maybe the crypto improvements can solve the security issue why multi-client repos are discouraged.

The caching is another reason why multi-client is sub-optimal currently, especially if you have many archives in the repo (== many files in chunks.archive.d or many archives to fetch).

@FabioPedretti the files cache is a purely local per-machine thing. The multi-client cache coherency issue needing cache resync is the chunks index (which contains information about the chunks in the repository, like presence, size, refcount).