document pull-like operation

[ThomasWaldmann]

this is a FAQ (by people who have firewalls or want it for other reasons) and some people are evaluating setups with ssh -R (see some posts in #36).

this issue is to collect such setups and if evaluated successfully, add it to the documentation.

note: the debian/ubuntu package description says borg only supports push, maybe that can be removed after this ticket is closed.

so, if you successfully run a pull-like setup, the best thing you can do is to make a pull request that closes this ticket.

---

💰 there is a bounty for this

Note: to collect the bounty you need to run a reliable pull-like setup, do a pull request for our documentation, documenting the pull-related parts of the setup.

[ThomasWaldmann]

A pull setup that does not involve ssh is to just mount the source filesystems on the machine that runs borg.

[textshell]

For the usecase where the normal push way is problematic because of firewalls etc.

From axion on irc (slightly edited and simplified, so all errors are likely mine):

repo=ssh://${USER}@localhost:${PULL_PORT}${REPO_PATH}/${host}
ssh -R ${PULL_PORT}:localhost:22 ${host}         \
  BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes \
  borg create ${repo}::${archive} /some/path

This way tunnels a ssh connection through an ssh connections so it does have some additional overhead.

Another way would be to use BORG_RSH and a pair of socat instances to avoid one layer of ssh encryption.

[ThomasWaldmann]

@textshell I don't think that BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes should be in there permanently, right?

Also, the repo=...{REPO_PATH}/${host} can be done that way, but is unrelated to pull-mode. Also, REPO_PATH there is rather the path that has the repos as subdirs, not the path of the repository itself.

[textshell]

@UNKNOWN_UNENCRYPTED it depends if PULL_PORT is always the same. If that can be arranged that yes, it should not be in there.

@repo yes that can be simplified. I only did limited editing from axion’s pastebin, but i didn’t want this to get lost again.

I think the BORG_RSH + socat way would be nicer anyway (no ssh in ssh, not dependency on sshd running on the backups server, etc), but a little more complex bash script.

[leaf-node]

While having documentation for this workaround is great, wouldn't it be better to add this functionality to borg itself? This kind of syntax would be awesome:

$ borg create /path/to/repo::example.com-now user@example.com:/

[cwebber]

I agree with @sudoman; it would be useful to know, are there architectural reasons this would be difficult? It feels like this would dramatically increase the number of scenarios for which Borg is a recommendable solution.

[textshell]

We first need to agree on a plan. For example i don‘t like overloading the directory argument on borg create with additional magic. I personally tend to a new sub command.

(Note: server is where the repo is, client is the remote system where the to be backuped data is)

Also as this is not the main use case for borg i think the design should minimize the changes needed. Thus i think the stdin, stdout and stderr of the ssh session should be used for the ui of the borg client on the remote system not for data transfer so that all interaction still works as expected. Thus the repository communication would need to be tunneled with an additional unix socket forward. I’m not sure what to do about borg serve’s stderr. Maybe it‘s ok to just (implicitly) splice that in on the server side.

One way to implement borg pull would be to create unix socket and listen to it, ssh to the to be backuped system to run borg with special options to create that tell it the needed unix socket to connect. The server would then wait for an connection on the unix socket, dup2 these to stdin, stdout and invoke the RepositoryServer.

Still open is how key management is even supposed to work in this scenario. Maybe mandate a keyfile on the client in the default location?
We also need to ensure that there is a good way to secure this with the usual forced command stuff.

This would need minimal changes to the main borg code:

• RemoteRepository would need to get an option to connect to a unix domain socket via a new option in create.
• A new pull command needs to be implemented that does the initial setup and then chains into borg serve.

Of course this still requires a borg executable on the client.

So it doesn‘t need any architectural changes but is a lot fiddling with external ssh process interaction and the os module. So in the end i think it‘s a task that is doable for anyone with sufficient motivation and decent python skills.