- evaluate modern chunking algorithms (fastcdc, ultracdc, ...)
- implement chunking algorithms for usage within borg (chunker written in Cython or from suitable, well-maintained existing external package, tests, interface with existing borg code)
- evaluate deduplication and throughput performance
- security: avoid fingerprinting and other attacks on the chunker
- rip out the borg.chunkers package from borg and create a borgchunk separate project with own tests and documentation
This task does not require much knowledge about the borg code base, but will require evaluating research papers, Cython coding, doing practical experiments, etc. - some math/crypto/security/algorithms background is useful here.
I recently converted our buzhash(32) code from C to Cython, then added a quite similar buzhash64 chunker (also Cython). So all code now is in Cython (some also in Python), no C anymore.
This task does not require much knowledge about the borg code base, but will require evaluating research papers, Cython coding, doing practical experiments, etc. - some math/crypto/security/algorithms background is useful here.
I recently converted our buzhash(32) code from C to Cython, then added a quite similar buzhash64 chunker (also Cython). So all code now is in Cython (some also in Python), no C anymore.