Multithreading

[ThomasWaldmann]

I started some experimental multithreading code there:

https://github.com/thomaswaldmann/borg/tree/multithreading

especially:

ThomasWaldmann@240a27a

From Python c-api docs:
"the standard zlib and hashlib modules release the GIL when compressing or hashing data."
So current compression and hashing should be ok for good multithreading, as well as python's I/O (it releases GIL before I/O ops).

Additionally, later changesets implement code to release the GIL there:

• chunker
• crypto
• lz4 compression

---

💰 there is a bounty for this

[ThomasWaldmann]

About the question in the commit comment about longer "user time" when running multithreaded compare to single-threaded:

The answer might be that this was a dual core cpu with hyperthreading. While this looks like 4 cores to the OS (and Python), when really using it like 4 cores, each of these cores might appear as a bit slower CPU than when just using a single core, which might explain the longer "user time" needs.

It still was quite faster on the wall clock, so we don't worry.

[ThomasWaldmann]

Some things are TODO there:

• remote repo tests are broken (and thus currently disabled)
• that delayer thread is strange, better solution?
• likely AES counter uniqueness is broken
• needs way more analysis / optimization
• fix the tests, they are quite broken / hang right now

Also, the bounty should be higher, this is quite a lot of work.

[enkore]

Just an idea

Traditional multi-threading (shared data, locking, queuing etc.) is often neither simple to develop nor test. I have been using ZeroMQ to effectively multi-thread things in a few situations now, and feel that it makes things much easier with essentially no relevant overhead (for inproc://), especially when using zerocopy.

I understand that in borg files are essentially processed by a pipeline of algorithms:

reading => chunking => hashing / encrypting => storage

This might lend itself well to the pipeline pattern in zmq, e.g. every one of these stages is handled by one or more nodes (=threads). Work distribution, queuing etc. are handled by zmq transparently.

This zmq approach to multithreading is not unlike microservices. It would probably mean extensive refactoring for Borg. OTOH it would lead to Borg scaling very well with available CPU resources.

[ThomasWaldmann]

Interesting idea. Can you say what the pros of this approach are compared to Python's Queue module (which is in stdlib and threadsafe)? The Queue module is what the multithreading code uses currently.

[enkore]

Ah I see how you approached it, it is conceptually already very close. I also see the "delayer problem" now.

On a technical level the zmq approach is language agnostic, so it's [it can be] straightforward to implement entire stages in C with no Python/GIL involvement at all (and the ability to still do this zerocopy: separate, pure C-thread runs a node in the same process, so inproc:// can be used between Python and C parts, without ever copying the data).

[stilsch]

Regarding compression:
We are using https://github.com/madler/pigz for parallel compressing of petabyte of archives. Maybe it could help here. We never had an issue - and the performance on multicore systems is awsome!

[ThomasWaldmann]

@schuft69 please see the ticket about compression about why this is not that simple in case of borg.