Skip to content

feat!: refactor packer and optimize memory allocations - #533

Merged
aawsome merged 1 commit into
mainfrom
refactor-packer
Jul 24, 2026
Merged

aawsome merged 1 commit into
mainfrom
refactor-packer

Conversation

@aawsome

@aawsome aawsome commented Jul 23, 2026 •

Copy link
Copy Markdown
Member

This PR refactors the packer and the way packfiles are handled in rustic:

  • Introduces and uses a BasicPacker which only provides the logic without doing any I/O itself
  • Packfiles are no longer collected as a single BytesMut in memory, but as BytesList which equals Vec<Bytes>. This reduces allocations and unnecessary memory copies (but is a breaking change w.r.t the backend API).
  • fixes the memory allocation used for the pack header. By accident, here again the total pack size was allocated. This fix should reduce the total needed memory especially when using large packfiles
  • this PR also fixes some new clippy lints (according to the given suggestions)

@codecov

codecov Bot commented Jul 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.26277% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.7%. Comparing base (82f32a4) to head (8cf2d88).

Files with missing lines Patch % Lines
crates/core/src/blob/packer.rs 77.1% 13 Missing ⚠️
crates/backend/src/rest.rs 0.0% 9 Missing ⚠️
crates/core/src/backend.rs 73.6% 5 Missing ⚠️
crates/core/src/blob/tree.rs 76.4% 4 Missing ⚠️
crates/backend/src/opendal.rs 0.0% 3 Missing ⚠️
crates/backend/src/rclone.rs 0.0% 1 Missing ⚠️
crates/core/src/backend/cache.rs 92.3% 1 Missing ⚠️
crates/core/src/backend/dry_run.rs 50.0% 1 Missing ⚠️
crates/core/src/backend/warm_up.rs 0.0% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
crates/backend/src/local.rs 41.4% <100.0%> (-1.1%) ⬇️
crates/core/src/backend/decrypt.rs 48.8% <100.0%> (+0.7%) ⬆️
crates/core/src/backend/hotcold.rs 57.1% <100.0%> (-7.2%) ⬇️
crates/core/src/commands/repair/hotcold.rs 65.1% <100.0%> (-1.2%) ⬇️
crates/core/src/crypto/aespoly1305.rs 85.7% <ø> (ø)
crates/core/src/repofile/packfile.rs 65.7% <100.0%> (-5.3%) ⬇️
crates/backend/src/rclone.rs 8.7% <0.0%> (ø)
crates/core/src/backend/cache.rs 57.1% <92.3%> (-1.8%) ⬇️
crates/core/src/backend/dry_run.rs 11.1% <50.0%> (ø)
crates/core/src/backend/warm_up.rs 0.0% <0.0%> (ø)
... and 5 more

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@aawsome aawsome changed the title feat: refactor packer and optimize memory allocations feat!: refactor packer and optimize memory allocations Jul 24, 2026
@aawsome
aawsome merged commit 4c9e1cb into main Jul 24, 2026
25 checks passed
@aawsome
aawsome deleted the refactor-packer branch July 24, 2026 10:28
@rustic-release-plz rustic-release-plz Bot mentioned this pull request Jul 24, 2026
pull Bot pushed a commit to ChecksumDev/rustic_core that referenced this pull request Aug 16, 2026
## 🤖 New release

* `rustic_core`: 0.12.0 -> 0.13.0 (✓ API compatible changes)
* `rustic_backend`: 0.6.2 -> 0.7.0 (✓ API compatible changes)
* `rustic_testing`: 0.4.1 -> 0.5.0 (✓ API compatible changes)
* `rustic_config`: 0.2.4 -> 0.2.5 (✓ API compatible changes)

<details><summary><i><b>Changelog</b></i></summary><p>

## `rustic_core`

<blockquote>

##
[0.13.0](rustic-rs/rustic_core@rustic_core-v0.12.0...rustic_core-v0.13.0)
- 2026-08-16

### Added

- stdio protocol between warmup program and rustic
([#1782](https://github.com/rustic-rs/rustic_core/pull/1782))
([rustic-rs#524](rustic-rs#524))
- implement some exclude options for opendal source
([rustic-rs#535](rustic-rs#535))
- Add sparse file support for restore
([rustic-rs#530](rustic-rs#530))
- [**breaking**] refactor packer and optimize memory allocations
([rustic-rs#533](rustic-rs#533))

### Fixed

- read full packs from cold backend in hot/cold case
([rustic-rs#542](rustic-rs#542))
- use no parent for stdin source
([rustic-rs#536](rustic-rs#536))

### Other

- *(deps)* dependency updates
([rustic-rs#541](rustic-rs#541))
- *(deps)* bump opendal to 0.58.1
([rustic-rs#539](rustic-rs#539))
- reduce memory usage for index
([rustic-rs#537](rustic-rs#537))
</blockquote>

## `rustic_backend`

<blockquote>

##
[0.7.0](rustic-rs/rustic_core@rustic_backend-v0.6.2...rustic_backend-v0.7.0)
- 2026-08-16

### Added

- implement some exclude options for opendal source
([rustic-rs#535](rustic-rs#535))
- [**breaking**] refactor packer and optimize memory allocations
([rustic-rs#533](rustic-rs#533))

### Other

- *(deps)* bump opendal to 0.58.1
([rustic-rs#539](rustic-rs#539))
</blockquote>

## `rustic_testing`

<blockquote>

##
[0.5.0](rustic-rs/rustic_core@rustic_testing-v0.4.1...rustic_testing-v0.5.0)
- 2026-08-16

### Added

- [**breaking**] refactor packer and optimize memory allocations
([rustic-rs#533](rustic-rs#533))

### Other

- *(deps)* dependency updates
([rustic-rs#541](rustic-rs#541))
</blockquote>

## `rustic_config`

<blockquote>

##
[0.2.5](rustic-rs/rustic_core@rustic_config-v0.2.4...rustic_config-v0.2.5)
- 2026-08-16

### Other

- *(deps)* bump opendal to 0.58.1
([rustic-rs#539](rustic-rs#539))
</blockquote>


</p></details>

---
This PR was generated with
[release-plz](https://github.com/release-plz/release-plz/).

Co-authored-by: rustic-release-plz[bot] <182542030+rustic-release-plz[bot]@users.noreply.github.com>
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
rustic 0.11.4

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>Hello rusticeans,

we are happy to release rustic 0.11.4! This release features sparse file support for `restore` and adds glob includes/excludes to backups from openDAL sources.
Also better progress indication for warm-up has been added and some memory usage optimizations have been landed - besides many other minor bug fixes.
Thanks a lot to everybody who contributed!

# Changelog

### Added

- Add sparse file support for restore ([#530](rustic-rs/rustic_core#530))
- support exclusion on the opendal source ([#1812](rustic-rs/rustic#1812), [#535](rustic-rs/rustic_core#535))
- stdio protocol between warmup program and rustic ([#1782](https://github.com/rustic-rs/rustic_core/pull/1782), [#524](rustic-rs/rustic_core#524))

### Fixed

- read full packs from cold backend in hot/cold case ([#542](rustic-rs/rustic_core#542), [#1868](rustic-rs/rustic#1868))
- use no parent for stdin source ([#536](rustic-rs/rustic_core#536))
- link jemallocator-global so the jemallocator feature applies ([#1821](rustic-rs/rustic#1821))
- treat unchanged symlinks as identical in diff ([#1829](rustic-rs/rustic#1829))
- keep completions independent from profiles ([#1830](rustic-rs/rustic#1830))
- allow scrolling long read-only TUI text ([#1851](rustic-rs/rustic#1851))
- record CLI version in snapshot metadata ([#1803](rustic-rs/rustic#1803))
- read stdin-filename from config file ([#1813](rustic-rs/rustic#1813))
- deny unknown forget options in config profile ([#1810](rustic-rs/rustic#1810))
- set correct UID/GID for dump into tar ([#1801](rustic-rs/rustic#1801))

### Other

- refactor packer and optimize memory allocations ([#533](rustic-rs/rustic_core#533))
- reduce memory usage for index ([#537](rustic-rs/rustic_core#537))
- Add pCloud service example config ([#1775](rustic-rs/rustic#1775))
- Update README ([#1783](rustic-rs/rustic#1783), [#1785](rustic-rs/rustic#1785))
- update dependencies ([#1882](rustic-rs/rustic#1882), [#1854](rustic-rs/rustic#1854), [#541](rustic-rs/rustic_core#541), [#539](https://github.com/rustic-rs/rustic_core/pull/539))</pre>
  <p>View the full release notes at <a href="https://github.com/rustic-rs/rustic/releases/tag/v0.11.4">https://github.com/rustic-rs/rustic/releases/tag/v0.11.4</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!17083
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant