Skip to content

zlib: improve zstd decoding across chunk boundaries - #65865

Closed
jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/zlib-decode-concat-skippable
Closed

zlib: improve zstd decoding across chunk boundaries#65865
jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/zlib-decode-concat-skippable

Conversation

@jasnell

@jasnell jasnell commented Sep 6, 2026

Copy link
Copy Markdown
Member

Discovered when testing some other changes that zstd was not handling certain chunk boundaries very well. This fixes it.

  • Decodes concatenated and skippable zstd frames across all chunk boundaries.
  • Preserves rejectGarbageAfterEnd semantics.
  • Reports corruption in subsequent frames.
  • Handles trailing partial frame identifiers consistently.
  • Covers classic sync/async and iterable APIs.
  • Updated documentation and internal typings.

Technically a behavior change but zstd is still experimental.

@jasnell
zlib: improve zstd decoding across chunk boundaries
461b2a7 
Discovered when testing some other changes that zstd was not
handling certain chunk boundaries very well. This fixes it.

- Decodes concatenated and skippable zstd frames across all chunk boundaries.
- Preserves rejectGarbageAfterEnd semantics.
- Reports corruption in subsequent frames.
- Handles trailing partial frame identifiers consistently.
- Covers classic sync/async and iterable APIs.
- Updated documentation and internal typings.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. typings Issues and PRs related to internal TypeScript declarations. zlib Issues and PRs related to the zlib module and its compression dependencies. labels Sep 6, 2026
@nodejs-github-bot

This comment was marked as outdated.

Sign in to view
@codecov

codecov Bot commented Sep 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.46341% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.19%. Comparing base (8af7545) to head (461b2a7).
⚠️ Report is 41 commits behind head on main.

Files with missing lines Patch % Lines
src/node_zlib.cc 90.41% 2 Missing and 5 partials ⚠️
Additional details and impacted files 
@@           Coverage Diff           @@
##             main   #65865   +/-   ##
=======================================
  Coverage   90.18%   90.19%           
=======================================
  Files         771      771           
  Lines      264619   264687   +68     
  Branches    50231    50256   +25     
=======================================
+ Hits       238653   238735   +82     
+ Misses      16961    16952    -9     
+ Partials     9005     9000    -5
Files with missing lines Coverage Δ
lib/zlib.js 98.15% <100.00%> (+<0.01%) ⬆️
src/node_zlib.cc 79.91% <90.41%> (+0.72%) ⬆️

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nodejs-github-bot

nodejs-github-bot commented Sep 7, 2026 •
edited by jasnell
Loading

Copy link
Copy Markdown
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/77141/ 💚

@jasnell
jasnell requested review from panva and trivikr September 8, 2026 00:23
@trivikr

trivikr commented Sep 8, 2026

Copy link
Copy Markdown
Member

The lint-commit-message is failing

Error: not ok 5 line-length: Line should be <= 72 columns.
...
          '- Decodes concatenated and skippable zstd frames across all chunk boundaries.',

@jasnell

jasnell commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

Yes, I'll fix the lint issue in the commit message when landing.

@jasnell jasnell mentioned this pull request Sep 8, 2026
Closed
@trivikr trivikr added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Sep 8, 2026
jasnell added a commit that referenced this pull request Sep 8, 2026
@jasnell
zlib: improve zstd decoding across chunk boundaries
7055c48 
Discovered when testing some other changes that zstd was not
handling certain chunk boundaries very well. This fixes it.

- Decodes concatenated and skippable zstd frames across all
  chunk boundaries.
- Preserves rejectGarbageAfterEnd semantics.
- Reports corruption in subsequent frames.
- Handles trailing partial frame identifiers consistently.
- Covers classic sync/async and iterable APIs.
- Updated documentation and internal typings.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: #65865
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
@jasnell

jasnell commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

Landed in 7055c48

@jasnell jasnell closed this Sep 8, 2026
aduh95 pushed a commit that referenced this pull request Sep 16, 2026
@jasnell @aduh95
zlib: improve zstd decoding across chunk boundaries
678ff35 
Discovered when testing some other changes that zstd was not
handling certain chunk boundaries very well. This fixes it.

- Decodes concatenated and skippable zstd frames across all
  chunk boundaries.
- Preserves rejectGarbageAfterEnd semantics.
- Reports corruption in subsequent frames.
- Handles trailing partial frame identifiers consistently.
- Covers classic sync/async and iterable APIs.
- Updated documentation and internal typings.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: #65865
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
@trotzig trotzig mentioned this pull request Sep 17, 2026
Open
@github-actions github-actions Bot mentioned this pull request Sep 20, 2026
Merged
aduh95 pushed a commit that referenced this pull request Sep 22, 2026
@jasnell @aduh95
zlib: improve zstd decoding across chunk boundaries
d7e7308 
Discovered when testing some other changes that zstd was not
handling certain chunk boundaries very well. This fixes it.

- Decodes concatenated and skippable zstd frames across all
  chunk boundaries.
- Preserves rejectGarbageAfterEnd semantics.
- Reports corruption in subsequent frames.
- Handles trailing partial frame identifiers consistently.
- Covers classic sync/async and iterable APIs.
- Updated documentation and internal typings.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
PR-URL: #65865
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
This was referenced Sep 27, 2026
Closed
Open
@aduh95 aduh95 mentioned this pull request Oct 13, 2026
Merged
This was referenced Oct 19, 2026
Merged
Merged
andimial pushed a commit to andimial/mneme that referenced this pull request Oct 20, 2026
@modusensus
test(session-usage): 多帧反证改数帧头，不再依赖 Node 的 zstd 整块解语义
5310e1b 
原反证断言 zstdDecompressSync(buf) 只能拿到第一帧，依赖的是「整块解只吐第一帧」这一
Node 旧行为。24.22.0 起解码跨 chunk 边界也能解出所有帧（nodejs/node#65865），断言在
legacy 腿（装 latest 24.x）上过期，main 下次 runner 升到 24.22.0 就会红。

改成数 zstd magic 帧头：既与产品代码 orEachFrame 的切帧判据同源，又不绑定解码器语义。
第二行断言用 stat.frames + stat.failed——帧头数 = 成功数 + 失败数 是实现契约，而
「帧头数 = 成功数」只在 failed 恒为 0 时成立，是巧合不是规矩。

已知局限写进注释：这条锁的是「切帧判据仍是 magic」，不是碰撞免疫。payload 里撞上
magic 时计数器与解码器会一起错（半片也解得出来），实测 frames 同样变成 4，这条测不出来。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@panva panva panva approved these changes
@trivikr trivikr trivikr approved these changes

Assignees

No one assigned

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. typings Issues and PRs related to internal TypeScript declarations. zlib Issues and PRs related to the zlib module and its compression dependencies.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jasnell @nodejs-github-bot @trivikr @panva