Reduce the size of the Linux standalone binaries

[ThomasWaldmann]

The Linux standalone binaries are about 70 MiB as a single file and unpack to 152 MiB (the single-file variant does that in a temp dir at every start). The FreeBSD and macOS binaries are 40-42 MiB. Most of the difference is avoidable.

Measured on the borg-linux-glibc239-x86_64-gh artifact of CI run 34374266883 (PR #10344); the borg-linux-glibc243-x86_64-gh binary of 2.0.0b24 has the same size (70.2 MiB), so this is not specific to the new job.

Where the size goes

item	unpacked	in the compressed binary	note
debug info + symbol tables in the 97 bundled shared libraries	~74 MiB	~26 MiB	Python's default CFLAGS contain -g, and scripts/borg.exe.spec has strip=False. libpython is 30.5 MiB (5.7 MiB stripped), borg's own Cython modules 22 MiB, the stdlib extension modules 21 MiB, then pyfuse3, yaml, msgpack, ...
botocore AWS service models (botocore/data)	23 MiB	~14.5 MiB	435 entries, already gzip-compressed JSON, so they hardly shrink. borgstore's S3 backend only creates an s3 client; it needs data/s3, endpoints.json, partitions.json, sdk-default-configuration.json, _retry.json: about 0.3 MiB compressed
pure-Python code archive inside borg.exe (PYZ)	14 MiB	14 MiB	textual, rich, paramiko, botocore's python code, borg itself
cryptography's Rust extension (_rust.abi3.so)	13.7 MiB	4.4 MiB	pulled in by paramiko (sftp extra); the manylinux wheel statically links its own copy of OpenSSL, so the binary contains two OpenSSLs
libcrypto	6.9 MiB	~2.5 MiB	real

Options

1. Strip the shared libraries on Linux (strip=True in the spec for Linux, or build Python and the extensions without -g).

   • Measured: 152 -> 78 MiB unpacked, 69.6 -> 43.4 MiB compressed (llvm-strip --strip-all on every .so, not on borg.exe, which carries the appended archive; then the same tar czf). A stripped bundle was tried on Debian 13: repo-create with aes256-ocb, create, list, check, extract all fine.
   • Pro: by far the biggest win, halves the unpack size and thus the single-file startup cost, no functional change. Python tracebacks are unaffected.
   • Con: no C-level symbols for gdb / core dumps of the frozen binary. The spec is shared by all platforms, so this should be conditional (strip=sys.platform == "linux"): on macOS, stripping interferes with code signing, and the Windows/MSYS2 build is a different story. Building without -g instead of stripping only works for the from-source Python of the glibc239 job, not for the setup-python builds of the glibc243 job.

2. Bundle only the botocore service models borg needs (filter a.datas in the spec, or a custom hook overriding the botocore hook): s3, sts (assume-role profiles), plus the top-level json files.

   • Saves about 14 MiB compressed / 23 MiB unpacked.
   • Pro: second biggest win, trivial to implement in the spec.
   • Con: has to be kept in sync with what borgstore uses; a missing model shows up as an obscure botocore error at runtime, so the S3 test in CI should cover it. Some credential providers (SSO, web identity) would need more models (sso, sso-oidc).

3. Build cryptography from source against the bundled OpenSSL instead of using the manylinux wheel.

   • Saves an estimated 3-5 MiB compressed (the vendored OpenSSL inside _rust.abi3.so).
   • Pro: one OpenSSL in the binary instead of two, i.e. one thing to update on an OpenSSL CVE.
   • Con: needs a Rust toolchain in the job, several minutes of build time, and the cryptography build against a non-vendored OpenSSL is more fragile across versions. Poor ratio of savings to effort.

4. Bundle mfusepy instead of pyfuse3 in the Linux binaries.

   • Saves about 1 MiB compressed (pyfuse3's extension and the bundled libfuse3).
   • Pro: no bundled libfuse, so the binary works with the system's libfuse2 or libfuse3 and matches the default BORG_FUSE_IMPL order (mfusepy first).
   • Con: mfusepy is newer and less battle-tested (see hlfuse: pass the libfuse options to mfusepy as keyword arguments, fix getattr with a file handle #10340); small saving.

5. Drop extras from the binary (cockpit: textual + rich; s3: boto3/botocore; sftp: paramiko + cryptography + pynacl), or offer a "lite" binary next to the full one.

   • Pro: the only way to get well below 30 MiB.
   • Con: loses features, or doubles the release matrix and the user confusion. Not recommended as the default.

6. .tar.xz instead of .tgz for the single-directory variant.

   • Pro: 20-30 % smaller download.
   • Con: renames a release asset, slower to unpack, does nothing for the single-file binary or for the unpack size.

7. UPX: the spec has upx=True for the single-file binary, but UPX is not installed on the runners, so it is a no-op today.

   • Con: on top of PyInstaller's own zlib compression it would gain little, and UPX-packed binaries trigger AV false positives and start slower. Not recommended; the flag could be removed for clarity.

Options 1 and 2 together would put the Linux binaries at roughly 29 MiB single-file / 55 MiB unpacked, below the current FreeBSD and macOS binaries, without losing any feature.

🤖 Generated with Claude Code

[ThomasWaldmann]

fixed by #10346 and #10347.