try parallel ops using --bypass-lock

[ThomasWaldmann]

borg usually does not let you do multiple operations in parallel. it blocks that using an exclusive lock on the repository.

there is a (potentially dangerous) hack with --bypass-lock.

for some situations, it might actually work ok without leading to problems.

practically try:

• borg create repo::archive lotsofdata plus borg mount repo
• borg mount repo plus borg create repo::archive lotsofdata
• (and other read-only operations like borg list)

Do not try multiple repo-writing ops in parallel!
Do not use a production repo for experiments, just create a temporary one!

[laur89]

Could the --bypass-lock opt allow read-only operations, but bail if concurrent write options are attempted?

[ThomasWaldmann]

No, it's a hack and not a enhanced locking system. That might come later, but don't hold your breath.

[zecKiLL]

practically try:

* `borg mount repo` plus `borg create repo::archive lotsofdata`

Hi Thomas,

Could you please confirm if this would work safely? I am using the latest v.1.2.0

Thank you in advance.

Cheers,
KC

[ThomasWaldmann]

You shouldn't use that, see the docs.

[zecKiLL]

You shouldn't use that, see the docs.

Hi Thomas,

Thank you very much for your quick reply... I am glad to have your confirmation that this operation being un-safe, even though I have tested it and seemingly nothing was broken.

I did some more looking while I was waiting:

1. Set env for BORG_BASE_DIR to a dedicated path: ~/test/base
2. Init and Perform 1st backup run at: ~/test/repo
3. Make full copy of ~/test to ~/test-original
4. Perform 2nd backup run (with more files than 1st run) at: ~/test/repo
5. Run a diff -qr ~/test-original ~/test

The results:

Files test-orig/base/.cache/borg/a78...664/chunks and test/base/.cache/borg/a78...664/chunks differ
Files test-orig/base/.cache/borg/a78...664/config and test/base/.cache/borg/a78...664/config differ
Files test-orig/base/.cache/borg/a78...664/files and test/base/.cache/borg/a78...664/files differ
Files test-orig/base/.config/borg/security/a78...664/manifest-timestamp and test/base/.config/borg/security/a78...664/manifest-timestamp differ
Files test-orig/base/.config/borg/security/a78...664/nonce and test/base/.config/borg/security/a78...664/nonce differ
Only in test/repo/data/0: 24
Only in test/repo/data/0: 25
Only in test/repo/data/0: 26
Only in test/repo/data/0: 27
Only in test/repo/data/0: 28
Only in test-orig/repo: hints.23
Only in test/repo: hints.28
Only in test-orig/repo: index.23
Only in test/repo: index.28
Only in test-orig/repo: integrity.23
Only in test/repo: integrity.28
Files test-orig/repo/nonce and test/repo/nonce differ

Apart from the new segment files and updated hints+index+integrity+nonce in repo/data after the 2nd run, it seems that there were only a handful of metadata files in BORG_BASE_DIR got updated. The old data segments were all "atomic" to the 1st backup and nothing got changed. I believe this is due to v1.2.0 now run as append-mode by default, correct?

Let's say if I want to mount the 1st backup as read-only (with --bypass-lock) while allowing the Repo continued to be worked on for subsequent backup runs (for "create" or other read-only operations only).

Would it be safe if I just make a copy of the entire BORG_BASE_DIR, along with the hints+index+integrity+nonce files from 1st run, stack them with overylayfs onto the same repo/data tree (reusing the same repo data store), essentially creating a snapshot of the repo (while allowing additional repo/data files to be added), then run "borg mount --bypass-lock) with a separate env vars of BORG_BASE_DIR pointing to the old BORG_BASE_DIR copy?

Sorry if it sounds convoluted... because it is :~)

Thank you again.

Cheers,
KC