Skip to content

fix(events): serialize websocket delivery and preserve reconnects - #135

Open
cesar-carlos wants to merge 2 commits into
evolution-foundation:mainfrom
cesar-carlos:fix/websocket-concurrent-write
Open

cesar-carlos wants to merge 2 commits into
evolution-foundation:mainfrom
cesar-carlos:fix/websocket-concurrent-write

Conversation

@cesar-carlos

@cesar-carlos cesar-carlos commented Jul 27, 2026 •

Copy link
Copy Markdown

Concurrent event dispatch could panic while writing WebSocket frames. Reconnecting an instance also left the old socket alive, whose eventual cleanup could remove the replacement. This change gives each physical socket one writer, closes superseded or failed sockets, and keeps registration independent of network writes.

Closes #99

Changes

  • Serialize JSON writes and their deadlines with one mutex per physical connection, including duplicate/shared instance and broadcast registrations. Preserve the existing lowercase queue and string payload envelope.
  • Snapshot recipients before network I/O. Each write has a 10-second deadline; failures close and unregister that exact connection while still attempting healthy recipients. Return joined errors for partial delivery without retries.
  • Keep the read loop in ServeWs, process control frames without buffering complete incoming messages, and clean up by connection identity so an old handler cannot remove a replacement.
  • Continue to the configured webhook after a WebSocket failure; do not retry a partially delivered WebSocket event. Omit submitted tokens from WebSocket authentication logs and skip Gin's automatic access logs for the query-authenticated WebSocket route.
  • Document connection replacement, deadlines, best-effort delivery, and the existing global-key/instance UUID requirements for /ws.

The writer and timeout handling follow Gorilla WebSocket v1.5.3's concurrency and deadline contracts.

Validation

Executed in Docker with Go 1.25.0, CGO, native image libraries, FFmpeg and Poppler:

  • go build ./...
  • go vet ./...
  • go test -timeout 5m ./...
  • go test -race -timeout 5m ./...
  • git diff --check

Regression tests verify complete delivery and unique payloads under concurrent writes, duplicate/shared registration, instance isolation, healthy recipients after failures, socket replacement, real upgrade/reconnect/disconnect cleanup, blocked writes, write deadlines, and independent webhook delivery without retries. Test workers and log files are closed.

The original PR head (a92ef00) reproduces hidden broadcast errors and webhook suppression with the added regression checks. The corrected implementation passes them. No real WhatsApp session, production database, license, or credentials are required or used; live HistorySync/receipt traffic was not exercised.

Summary by Sourcery

Harden WebSocket event delivery and reconnection handling while preserving payload compatibility and ensuring independent webhook delivery.

Bug Fixes:

  • Prevent concurrent WebSocket event writes from corrupting frames or panicking.
  • Ensure superseded, failed, and disconnected sockets are closed and removed without allowing stale cleanup to remove replacements.
  • Continue webhook delivery after WebSocket failures without retrying potentially partial WebSocket deliveries.

Enhancements:

  • Make WebSocket delivery best-effort across snapshotted recipients with per-connection serialization, deadlines, and partial-error reporting.
  • Preserve shared and duplicate registrations while isolating instance-targeted delivery and cleaning up connections by physical socket identity.
  • Process incoming WebSocket control traffic through the connection lifecycle while discarding client data messages.

Documentation:

  • Document global-key authentication, UUID-based instance connections, replacement behavior, write deadlines, cleanup, and best-effort delivery semantics for WebSockets.

Tests:

  • Add regression coverage for concurrent delivery, shared registrations, instance isolation, failed-recipient handling, reconnection cleanup, blocked writes, deadlines, and independent webhook delivery.

Chores:

  • Avoid exposing WebSocket authentication tokens in logs and omit automatic access logging for the query-authenticated WebSocket route.

gorilla/websocket allows only one concurrent writer. CallWebhook runs in
a goroutine per event, so Produce() could WriteJSON the same Conn from
multiple goroutines under load (HistorySync/Receipts) and panic, killing
the whole process.

Wrap each Conn in wsConn with a write mutex so RLock still protects the
client maps while writes are serialized per connection.

Closes evolution-foundation#99

Co-authored-by: Cursor <cursoragent@cursor.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @cesar-carlos, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@sourcery-ai

sourcery-ai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Reviewer's Guide

Introduces a per-connection wrapper around gorilla/websocket connections to serialize writes and adds race-detector tests that exercise concurrent instance-specific and broadcast websocket event delivery.

Sequence diagram for serialized websocket writes via wsConn

sequenceDiagram
    participant Producer as websocketProducer
    participant Map as clients_and_broadcast
    participant WsConn as wsConn
    participant WS as websocket_Conn

    Producer->>Map: Produce(queueName, payload, instanceID)
    Map-->>Producer: wsConn for instanceID
    Producer->>WsConn: writeJSON(message)
    activate WsConn
    WsConn->>WsConn: writeMu.Lock()
    WsConn->>WS: WriteJSON(message)
    WS-->>WsConn: error or nil
    WsConn->>WsConn: writeMu.Unlock()
    deactivate WsConn
    WsConn-->>Producer: error or nil
Loading

File-Level Changes

Change Details Files
Serialize all websocket writes per connection using a mutex-protected wrapper type to prevent concurrent writer panics while keeping map access concurrency as before.
  • Introduce wsConn struct holding *websocket.Conn plus a sync.Mutex and provide a writeJSON helper that locks around WriteJSON calls.
  • Change websocketProducer client maps and broadcast slices to store *wsConn instead of *websocket.Conn and update constructor accordingly.
  • Wrap new connections in wsConn when adding instance-specific and broadcast clients, and update removal logic to compare underlying *websocket.Conn.
  • Update Produce to call wsConn.writeJSON for both instance-specific and broadcast sends while still guarding client lookups with clientsMux RLock.
pkg/events/websocket/websocket_producer.go
Add concurrency-focused tests that validate websocketProducer handles many concurrent Produce calls for both instance-specific and broadcast connections without races or errors.
  • Add a helper logger factory using temporary directories to avoid external logging side effects in tests.
  • Add a dialTestWS helper that spins up an httptest server, upgrades to websocket, returns the server-side *websocket.Conn, and drains client reads.
  • Add TestProduceConcurrentWrites with subtests that create many goroutines issuing Produce calls against a single instance client and a single broadcast client, asserting no errors occur under concurrency.
pkg/events/websocket/websocket_producer_test.go

Assessment against linked issues

Issue Objective Addressed Explanation
#99 Prevent panic 'concurrent write to websocket connection' by serializing writes on each Gorilla WebSocket connection used by websocketProducer.Produce ✅
#99 Ensure both per-instance and broadcast WebSocket clients use the serialized-write mechanism when sending events under concurrent load ✅
#99 Add automated tests (including with -race) that exercise concurrent Produce calls for instance and broadcast clients to guard against regressions ✅

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

ecosb2b pushed a commit to ecosb2b/evo-go-v2 that referenced this pull request Aug 5, 2026
Merges the Athene fork, which sits 11 commits ahead of the same upstream base
and had already solved — in production — the three problems this fork was
working through, plus a batch of upstream PRs we had not picked up.

Only pkg/whatsmeow/service/whatsmeow.go conflicted (6 hunks); the other 63 files
merged cleanly. Every conflict was resolved in favour of the incoming version,
because each one was a place where both forks fixed the same bug and theirs is
the one we chose to keep:

- Connection leak: their getAuthContainer/sharedAuthContainer replaces our
  storeContainer/storeContainerHolder. Same fix, but theirs is a backport of
  upstream PR evolution-foundation#117, so it will converge with upstream instead of conflicting
  when that PR merges. It also caps SQLite at MaxOpenConns(1), which ours did
  not.
- Reconnection: their runtime supervisor replaces our EnableAutoReconnect=true.
  Ours stopped the automatic disconnect loop but left ReconnectClient reachable
  from the API endpoint and the send retry, so those paths could still spawn a
  second client. Their runtime token refuses to start a second runtime for an
  instance from any path, waits for the previous goroutine to exit, deduplicates
  concurrent reconnects, backs off exponentially with jitter, and probes to
  confirm the reconnect actually connected.
- Shared maps: ClientMapsMu closes the "fatal error: concurrent map writes"
  hole, which was still open on our side and kills the whole process rather than
  panicking a single request.

Also arriving with the merge: upstream PRs evolution-foundation#149, evolution-foundation#135, evolution-foundation#34, evolution-foundation#100, evolution-foundation#143, evolution-foundation#137,
evolution-foundation#120, evolution-foundation#130, evolution-foundation#151 and evolution-foundation#122; multi-webhook fan-out; POST /send/product;
POST /user/savecontact; GET /server/stats and the dashboard; the whatsmeow
update to 20260721; and removal of the 61 MB compiled binary that was tracked in
git.

Our GHCR workflow did not conflict and is preserved.

store_container_test.go was adapted to the incoming implementation: it now
exercises getAuthContainer, asserts against their MaxOpenConns of 20, and resets
sharedAuthContainer between tests, since that container is a package-level
global and would otherwise carry between tests and invalidate the pg_stat_activity
measurements. It is kept alongside their auth_container_retry_test.go because it
measures something theirs does not: the connection count the server actually
sees.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cesar-carlos cesar-carlos changed the title fix(events): serialize websocket writes per connection fix(events): serialize websocket delivery and preserve reconnects Oct 8, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sourcery assessment

Approved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic: concurrent write to websocket connection when sending WebSocket events (0.7.2)

1 participant