Skip to content

feat(session): expose EGFX compositing - #2120

Merged
Marc-André Moreau (mamoreau-devolutions) merged 3 commits into
Devolutions:masterfrom
willie:feat/session-composite-graphics
Oct 23, 2026
Merged

Marc-André Moreau (mamoreau-devolutions) merged 3 commits into
Devolutions:masterfrom
willie:feat/session-composite-graphics

Conversation

@willie

@willie Willie Abrams (willie) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

ActiveStage composites the EGFX output after each PDU. That only works when the GraphicsPipelineClient lives inside it.

Why run it somewhere else:

  • H.264 decoding can be slow (a busy GPU).
  • While it decodes, the session loop doesn't read the socket.
  • Audio waits behind video, and GNOME Remote Desktop drops audio older than 50 ms.
  • Running the pipeline on its own thread fixes that, but then nothing composites its output: the reset and composite steps were private.

What this adds:

  • composite_graphics_pipeline(gfx, image): follows a pending ResetGraphics, applies the finished output regions, and takes the RemoteApp window regions.
  • CompositedGraphics: whether it reset, the union region, each region, and the window updates.
  • ActiveStage calls it too, so its behavior doesn't change.
  • ironrdp-egfx is now a # public dependency, since GraphicsPipelineClient is in the signature. CompositedGraphics is #[non_exhaustive].

Using it off the session loop, the caller also:

  • registers a channel processor named ironrdp_egfx::CHANNEL_NAME that forwards each message to the pipeline;
  • sends the pipeline's replies (capabilities, frame acks) through the tunnel dvc_tunnel_for_channel names, or over TCP with ActiveStage::encode_dvc_messages;
  • shares the image with the session under a lock (bitmaps and the pointer write to it too);
  • paints the window updates into the RemoteApp windows they name, as ActiveStage returns them in WindowSurfaceUpdates;
  • remembers a reset until the next paint, and tracks damage itself.

The function's docs say the same.

🤖 Generated with Claude Code

@willie

Willie Abrams (willie) commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Validation at 31a3331:

  • New test composite_graphics_pipeline_follows_a_reset_then_paints_the_frame (ironrdp-testsuite-core, session::active_stage): a pipeline fed ResetGraphics, CreateSurface, MapSurfaceToOutput and a solid fill in one frame. Checks the image is resized, the fill's pixels land, the region covers them, and a second call returns nothing. It fails with either the reset or the compositing step removed.
  • New test active_stage_widens_after_a_reset_and_records_damage: a pipeline registered on DRDYNVC, fed through ActiveStage::process as X.224 frames. Checks the update after a reset covers the whole image and that a later fill is recorded as damage. It fails with the damage step removed. Removing the reset flag alone still passes: mapping a surface after the reset already outputs the whole surface.
  • cargo test -p ironrdp-testsuite-core: 1837 passed
  • RUSTDOCFLAGS=-D warnings cargo doc -p ironrdp-session: clean
  • cargo xtask check fmt, cargo xtask check lints, typos: clean

Where it's used: a macOS client I work on runs the pipeline on its own thread and calls this after each EGFX message. Against a headless GNOME Remote Desktop with 50 ms and 150 ms of artificial decode delay per update, all of a 17 s tone arrived; with the pipeline on the session loop, audio stopped after 3 s and the server dropped the connection.

Note

Human-tuned, LLM-assisted content.

@willie
Willie Abrams (willie) marked this pull request as ready for review October 13, 2026 13:36
Copilot AI balanced review requested due to automatic review settings October 13, 2026 13:36
@github-actions github-actions Bot added kind/protocol Affects RDP or related protocol behavior risk/high Substantial core public API impact, or fail-closed triage; needs maintainer-level scrutiny scope/core Touches the core architectural tier size/M Size: up to 449 counted lines and 10 files; exceeds S in either measure labels Oct 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The usage documentation incorrectly routes tunneled EGFX replies through the TCP-only encoder.

1 open finding
What changed in this PR

Exposes EGFX compositing so pipelines can run outside the session loop without blocking other channels.

Changes:

  • Adds public compositing results and helper API.
  • Reuses the helper within ActiveStage.
  • Adds integration coverage and marks EGFX as public.
File Description
crates/​ironrdp-session/​src/​active_stage.rs Adds and integrates the compositing API.
crates/​ironrdp-session/​src/​lib.rs Re-exports the new public API.
crates/​ironrdp-session/​Cargo.toml Marks EGFX as a public dependency.
crates/​ironrdp-testsuite-core/​tests/​session/​active_stage.rs Tests reset, compositing, and damage behavior.

🧠 Review effort: Balanced

Comment thread crates/ironrdp-session/src/active_stage.rs Outdated
@github-actions github-actions Bot added the needs-author-action The pull request author is the current next actor label Oct 13, 2026
@github-actions github-actions Bot removed the needs-author-action The pull request author is the current next actor label Oct 15, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #2120 extracts ActiveStage's private EGFX drain logic into a public composite_graphics_pipeline plus a #[non_exhaustive] CompositedGraphics result, making ironrdp-egfx a public dependency so off-loop callers (e.g., a dedicated decode thread) can composite without reimplementing reset/compositing subtleties. Verified behaviorally equivalent to the old inline path for all cases (no pipeline, reset only, updates only, both), including the Ok(None) channel-absent case. Two new tests cover the public path and the preserved ActiveStage path (reset widening, damage recording). No correctness or protocol defects found. Published one low-severity doc gap (docs omit the failure mode of compositing a pipeline ActiveStage still owns via the public get_dvc_mut), one low-severity control-flow style note on the rewritten drain method, and one low-severity test-fixture duplication note; rejected the suggestion to drop CompositedGraphics::region because the union helper is private and the field is …

  1. [code-compressor] Rewritten drain_graphics_pipeline chains Option combinators where a flat let-else reads smaller — low 🟡 — crates/ironrdp-session/src/active_stage.rs
    The rewrite replaces the old map(...).unwrap_or_default() with map(...).transpose()? plus a let-else, keeping the combinator chain only to thread the channel-absent case. Since get_dvc_mut already returns Option<DynamicChannelMut<T>>, the same behavior is shorter and flat: bind the channel with a let-else returning Ok(None), then call composite_graphics_pipeline(gfx.processor_mut(), image)? directly. Error propagation, the None return, and lifetimes are identical. Optional style refinement; the current form is correct.

Push a commit after addressing these findings. If no code change is needed, you may resolve inline threads and comment @github-actions review-ready to request human review.

Comment on lines +1025 to +1042
/// [`ActiveStage`] does this after each PDU for a [`GraphicsPipelineClient`]
/// registered on its dynamic channels. Call it directly for one that runs
/// elsewhere, such as on a thread of its own so that a slow H.264 decode
/// doesn't hold up the session's other channels. The caller then also:
///
/// - registers in its place a [`DvcClientProcessor`] named
/// [`ironrdp_egfx::CHANNEL_NAME`] that forwards each message to the
/// pipeline;
/// - sends the pipeline's replies (capabilities, frame acknowledgements):
/// wraps them with [`ironrdp_dvc::encode_dvc_messages`] for the channel,
/// then, if [`ActiveStage::dvc_tunnel_for_channel`] names a Soft-Sync
/// tunnel, sends each one's [`SvcMessage::encode_unframed_pdu`] through
/// it, and otherwise encodes them for TCP with
/// [`ActiveStage::encode_dvc_messages`];
/// - shares `image` with the session under a lock, since other updates
/// (bitmaps, the pointer) write to it too;
/// - tracks the regions itself: they don't reach
/// [`ActiveStage::take_damage_regions`].

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[skeptical] Docs omit the failure mode of compositing a pipeline that ActiveStage still owns — low 🟡 — ActiveStage::get_dvc_mut::<GraphicsPipelineClient>() is already public (active_stage.rs:493), so a caller can fetch the in-stage pipeline and pass it to the now-public composite_graphics_pipeline. That drains take_output_reset() and drain_output() before ActiveStage::process runs, so the in-stage drain sees no reset and no updates: graphics_output_needs_full_refresh is never set, widen_to_full_refresh never fires, and regions never reach take_damage_regions. The image is cleared and painted while the caller repaints only the returned region, leaving pre-reset content as silent visual corruption. The documented caller obligations cover the intended off-loop setup but do not warn that one pipeline must not be driven from both modes; a one-sentence doc guard (or debug-time detection) would close this.

Comment on lines +359 to +382
};
let fill = |frame_id, rect: ExclusiveRectangle| {
[
GfxPdu::StartFrame(StartFramePdu {
timestamp: Timestamp {
milliseconds: 0,
seconds: 0,
minutes: 0,
hours: 0,
},
frame_id,
}),
GfxPdu::SolidFill(SolidFillPdu {
surface_id: 1,
fill_pixel: Color {
b: 0x30,
g: 0x20,
r: 0x10,
xa: 0xFF,
},
rectangles: vec![rect],
}),
GfxPdu::EndFrame(EndFramePdu { frame_id }),
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[code-compressor] Two new tests rebuild identical EGFX PDU fixtures instead of sharing a helper — low 🟡 — Both new tests inline the same StartFrame/SolidFill/EndFrame triple, the all-zero Timestamp, the same fill color, and the same ResetGraphics/CreateSurface/MapSurfaceToOutput sequence; test 2 builds the fill triple via a local closure while test 1 repeats it inline. Hoisting fill (and optionally the surface-setup list) to file scope as small helpers returning Vec<GfxPdu> would cut roughly 40 lines and keep the fixture in one place. Test 2's gfx() frame wrapper is transport-specific and stays per-test.

@github-actions github-actions Bot added ai-reviewed/1 One automated review completed needs-author-action The pull request author is the current next actor labels Oct 15, 2026
ActiveStage composites the EGFX output after each PDU. That only works
when the GraphicsPipelineClient lives inside it.

Why run it somewhere else:
- H.264 decoding can be slow (a busy GPU).
- While it decodes, the session loop doesn't read the socket.
- Audio waits behind video, and GNOME Remote Desktop drops audio older
  than 50 ms.
- Running the pipeline on its own thread fixes that, but then nothing
  composites its output: the reset and composite steps were private.

What this adds:
- composite_graphics_pipeline(gfx, image): follows a pending
  ResetGraphics, then applies the finished output regions.
- CompositedGraphics: whether it reset, the union region, each region,
  and the RemoteApp window updates.
- ActiveStage calls it too, so its behavior doesn't change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The function's docs now say what running the pipeline outside
ActiveStage takes: a forwarding channel processor, sending the replies,
locking the image, painting the window updates, and tracking damage.
After a reset the caller repaints the whole image, as ActiveStage does.

ironrdp-egfx is marked public, since GraphicsPipelineClient is in the
signature, and CompositedGraphics is non_exhaustive without the unused
Default.

A test drives a registered pipeline through ActiveStage and checks the
widened update after a reset and the recorded damage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The steps for a caller of composite_graphics_pipeline sent the
pipeline's replies with ActiveStage::encode_dvc_messages, which always
encodes for TCP, so they went to the wrong transport once Soft-Sync
moved the channel onto a tunnel. Name the tunnel route, the
ironrdp_dvc::encode_dvc_messages step the replies need first, and that
a reset must be remembered until the next paint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot removed the needs-author-action The pull request author is the current next actor label Oct 23, 2026
@mamoreau-devolutions

Copy link
Copy Markdown
Contributor

Rebased onto master (8668111b) and force-pushed.
The previous head was 3c4c95c8.

What the rebase changed

  • feat(egfx)!: composite surfaces mapped to RemoteApp windows #2197 made drain_graphics_pipeline return the RemoteApp WindowUpdates too, and added ActiveStageOutput::WindowSurfaceUpdates.
    composite_graphics_pipeline now takes them as well (CompositedGraphics::window_updates), so ActiveStage still forwards them and an off-loop caller can paint them.
    They count against the compositor's shared byte budget, so a caller that never drained them would starve the surfaces and the cache.
  • ActiveStage::process gained a received_at argument on master, so the second test passes None for it now.
  • Cargo.toml takes master's dependency versions and keeps the # public marker on ironrdp-egfx.
  • New test: composite_graphics_pipeline_takes_the_window_updates.
  • The commits keep Willie as author.
    The two whose content changed carry a Co-authored-by trailer for the rebase.

Validation at 397fc69e

  • cargo xtask check fmt, lints, tests (whole workspace), locks and typos pass.
  • RUSTDOCFLAGS='-D warnings' cargo doc -p ironrdp-session --no-deps passes, so the intra-doc links resolve against current master.
  • The session::active_stage group runs 12 tests.

Lines changed against master

Added Removed
Production (ironrdp-session/src) 72 13
Tests (ironrdp-testsuite-core) 348 0
Other (ironrdp-session/Cargo.toml) 1 1
Total 421 14

32 of the added production lines are rustdoc.

Note

LLM-assisted content (no human feedback).

@mamoreau-devolutions
Marc-André Moreau (mamoreau-devolutions) merged commit c25dba1 into Devolutions:master Oct 23, 2026
58 of 59 checks passed
Anton Mostovoy (antonmos) added a commit to antonmos/IronRDP that referenced this pull request Oct 26, 2026
…ktop-scale-factor

* upstream/master: (92 commits)
  fix(server): advertise horizontal mouse wheel input (Devolutions#2236)
  fix(pdu): reject RSA moduli below 512 bits (Devolutions#2225)
  ci(ffi): verify generated bindings (Devolutions#2166)
  build(ffi)!: move to Diplomat 0.16.1-devo.2 (Devolutions#2165)
  build: make the client's video, camera and decoder crates publishable (Devolutions#2239)
  ci(release): release the newly published crates (Devolutions#2218)
  perf(graphics): replace the RLGR decoder's bit reader (Devolutions#2150)
  chore(graphics): add RemoteFX decode benchmarks (Devolutions#2149)
  fix(server)!: pass a GfxContext to the GfxServerFactory build methods (Devolutions#2104)
  fix(pdu): check remaining length when skipping padding and reading client core data (Devolutions#2130)
  fix(session): avoid hang on empty QOIZ updates (Devolutions#2147)
  feat(session): expose EGFX compositing (Devolutions#2120)
  fix(server): send RDP as the Demand Active source descriptor (Devolutions#2119)
  feat(activex)!: broaden mstscax compatibility (Devolutions#2211)
  feat(viewer): run RemoteApp programs in native windows (Devolutions#2201)
  fix(graphics): reject RLGR3 values larger than their remainder (Devolutions#2132)
  feat(egfx)!: composite surfaces mapped to RemoteApp windows (Devolutions#2197)
  feat: redirect USB devices over URBDRC with nusb and RemoteFX USB backends (Devolutions#2181)
  build: mark exposed dependencies as public (Devolutions#2213)
  chore(release): prepare for publishing (Devolutions#2212)
  ...

# Conflicts:
#	crates/ironrdp-testsuite-extra/tests/e2e.rs

This branch was previously deployed

1 inactive deployment
llm-providers — 397fc69e Deployed Oct 23, 2026 by mamoreau-devolutions via Classify pull request #7867
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-reviewed/1 One automated review completed kind/protocol Affects RDP or related protocol behavior risk/high Substantial core public API impact, or fail-closed triage; needs maintainer-level scrutiny scope/core Touches the core architectural tier size/M Size: up to 449 counted lines and 10 files; exceeds S in either measure

Development

Successfully merging this pull request may close these issues.

3 participants