Skip to content

RedisCluster: sharded Pub/Sub silently dropped after in-place slot migration — event-name mismatch (client emits sharded-channel-moved, cluster listens for server-sunsubscribe) #3311

Description

@dmonad

Description

Hi there, I'm having issues with channel resubscription on sharded channels. The below issue description is LLM generated, but reviewed and verified by me.

Bug

When a sharded Pub/Sub channel's hash slot is migrated in place between two live nodes (ordinary CLUSTER SETSLOT rebalancing, no node going down), RedisCluster permanently stops delivering messages for that channel. The client stays subscribed on the old owner — which no longer owns the slot — and is never resubscribed on the new owner. No error reaches the application.

The pub/sub docs say this is "handled automatically for you" with RedisCluster; it is not, for the in-place migration case.

Environment

  • redis / @redis/client: 6.0.0 (latest). Same code on current master.
  • Server: Valkey 7 (open-source Redis Cluster), 3 masters + 3 replicas.

Root cause — event-name mismatch

The emitter and the listener use different event names, so they never connect:

  • Emitter — on a server-initiated sharded unsubscribe push (sent by the migrating node when the slot changes hands), the commands queue invokes its onShardedChannelMoved callback, which in client/lib/client/index.ts (#initiateQueue) emits:
    (channel, listeners) => this.emit('sharded-channel-moved', channel, listeners)
  • Listener — the cluster's sharded Pub/Sub client in client/lib/cluster/cluster-slots.ts (#initiateShardedPubSubClient) listens for a name that is never emitted anywhere in the package:
    .on('server-sunsubscribe', async (channel, listeners) => {
      await this.rediscover(client);
      const redirectTo = await this.getShardedPubSubClient(channel);
      await redirectTo.extendPubSubChannelListeners(PUBSUB_TYPE.SHARDED, channel, listeners);
    })
    (The catch block in this method also has a typo: it emits 'sharded-shannel-moved-error' — "shannel".)

Because 'server-sunsubscribe' is never emitted and nothing handles 'sharded-channel-moved' internally, the rediscover-and-resubscribe logic never runs on an in-place migration.

RESUBSCRIBE_LISTENERS_EVENT → resubscribeAllPubSubListeners does handle SHARDED listeners, but it only fires on node removal / socket reconnect ('__MOVED') and on the Redis Enterprise "smigrated" push (#handleSmigrated) — not on an in-place slot move between two live nodes.

Reproduction (verified)

3-master OSS cluster. Subscribe with a RedisCluster client, then migrate the channel's slot to another live master:

import { createCluster } from 'redis'
const cluster = createCluster({ rootNodes: [{ url: 'redis://127.0.0.1:7000' }] })
await cluster.connect()

const channel = 'reprochan'            // slot 7061, owned by :7002
let received = 0
await cluster.sSubscribe(channel, () => received++)

Migrate slot 7061 from :7002 to :7000 (both stay up; a channel is not a key, so no MIGRATE step):

valkey-cli -p 7000 cluster setslot 7061 importing <node-id-7002>
valkey-cli -p 7002 cluster setslot 7061 migrating <node-id-7000>
valkey-cli -p 7000 cluster setslot 7061 node      <node-id-7000>
valkey-cli -p 7002 cluster setslot 7061 node      <node-id-7000>
# propagate the final SETSLOT NODE to every master

Then SPUBLISH reprochan "hi" on the new owner :7000.

Observed:

[BEFORE] SPUBLISH on :7002 -> 1 receiver; client received it          ✅
'sharded-channel-moved' EMITTED for channel="reprochan"
SHARDCHANNELS @old-owner :7002 -> ""        (subscription dropped)
SHARDCHANNELS @new-owner :7000 -> ""        (client NOT resubscribed here)
[AFTER]  SPUBLISH on :7000 -> 0 receivers; client received nothing    ❌

Expected: after migration the subscriber keeps receiving messages. Actual: delivery stops permanently, silently.

Suggested fix

Make emitter and listener agree — either have #initiateShardedPubSubClient listen for 'sharded-channel-moved' (the event actually emitted) or rename the emitted event to 'server-sunsubscribe'. Also fix the 'sharded-shannel-moved-error' typo. Note the related correctness problem when several sharded channels hash to different slots (a single SSUBSCRIBE with all of them returns CROSSSLOT) — resubscription should be grouped per slot (see #2902).

Workaround (verified)

Listen for 'sharded-channel-moved' on the underlying node pub/sub client(s) and resubscribe. Confirmed to recover delivery (subscription reappears on the new owner; SPUBLISH reaches 1 receiver again). Requires reaching into cluster internals, so it's fragile across versions:

for (const master of cluster.masters) {
  master.pubSub?.client?.on('sharded-channel-moved', ch => cluster.sSubscribe(ch, handler))
}

Related: #2902.

Node.js Version

No response

Redis Server Version

7.x (3+ clients)

Node Redis Version

6.0.0

Platform

No response

Logs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions