Repository navigation
[1.19] feat(pubsub/rabbitmq): support externally managed and consistent hash exchanges - #4552
Conversation
… exchanges The RabbitMQ pub/sub component always issues an active `exchange.declare` for a topic and can only declare one of the four built-in AMQP exchange types. That makes two topologies impossible today: 1. An exchange owned by something other than Dapr - for example the RabbitMQ Cluster Kubernetes Topology Operator, or Terraform. AMQP declaration is idempotent only when *every* property matches, so a pre-existing exchange whose properties differ from the component's defaults fails permanently with PRECONDITION_FAILED (406). 2. Partitioned ordering with horizontal scale-out, which on RabbitMQ is an `x-consistent-hash` exchange from the rabbitmq_consistent_hash_exchange plugin. The component's only in-the-box ordering control is `concurrency: single`, a single serial consumer. This adds: - `exchangeDeclareMode` (`declare` | `passive`, default `declare`). In `passive` mode the component asserts that the exchange exists via a passive `exchange.declare` and never creates or modifies it, so an external owner is the single source of truth for the topology. Since RabbitMQ ignores every argument but the name on a passive declare, an exchange of any kind - including plugin kinds - is usable this way. The mode applies to the topic exchange and to the dead letter exchange. - `x-consistent-hash` as a declarable `exchangeKind`, and acceptance of any broker-supported kind when `exchangeDeclareMode` is `passive`. - Validation of the binding key for consistent hash exchanges, which must be a positive integer bucket weight, reported as a terminal subscription error instead of an opaque channel failure. - Actionable messages for the two AMQP failures specific to externally managed topologies: PRECONDITION_FAILED now points at `exchangeDeclareMode: passive`, and a passive declare against a missing exchange says the component will not create it. The default behaviour is unchanged. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4552 +/- ##
==========================================
+ Coverage 31.94% 34.71% +2.77%
==========================================
Files 353 355 +2
Lines 47723 38956 -8767
==========================================
- Hits 15243 13525 -1718
+ Misses 31277 24152 -7125
- Partials 1203 1279 +76 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ertification tests
Extends the externally managed topology support to queues, and covers both
features with certification tests against a real broker.
The exchange-only knob was not enough. Where the RabbitMQ Cluster Kubernetes
Topology Operator is the source of truth it owns Queue and Binding resources
as well as Exchange resources, and the component declaring its own queue and
binding collides with it in exactly the same way.
- `queueDeclareMode` (`declare` | `passive`, default `declare`). In `passive`
mode the component asserts that the consumer queue - and the dead letter
queue, when `enableDeadLetter` is set - already exists, and leaves the
bindings to the external owner. Queue arguments become the owner's
responsibility.
It is deliberately independent of `exchangeDeclareMode`: an operator-owned
exchange is commonly paired with queues the component still declares,
because consumer queue names are only known at runtime. That combination is
what makes consistent hash scale-out work.
- Certification tests:
- `TestRabbitMQPassiveTopology` declares the exchange, queue and binding
out-of-band, runs a fully passive component against them, and asserts via
the management API that the set of exchanges and queues on the broker is
byte-for-byte unchanged afterwards - the component created nothing.
- `TestRabbitMQConsistentHashPartitioning` runs two independently scaled
consumers against an operator-owned x-consistent-hash exchange, publishes
a composite tenant/serial partition key per message, and asserts the three
properties that make this an alternative to concurrency=single: every key
is handled by exactly one consumer, the key space is spread across all of
them, and per-key order is preserved.
The certification broker now enables rabbitmq_consistent_hash_exchange via
a mounted enabled_plugins file.
Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
…paths Only PRECONDITION_FAILED, and NOT_FOUND while in passive mode, are rewritten into topology-ownership guidance. Pin that every other error - a non-AMQP error, an AMQP error of another code, and a NOT_FOUND raised while the component is the one declaring - reaches the caller untouched. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds externally managed RabbitMQ topology support and consistent-hash exchanges.
Changes:
- Adds passive exchange and queue declaration modes.
- Supports
x-consistent-hashexchanges with binding-weight validation. - Adds unit, integration, and certification coverage.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
pubsub/rabbitmq/rabbitmq.go |
Implements passive declarations, validation, and error decoration. |
pubsub/rabbitmq/metadata.go |
Parses new modes and exchange kind. |
pubsub/rabbitmq/metadata.yaml |
Documents new metadata options. |
pubsub/rabbitmq/rabbitmq_test.go |
Adds unit tests. |
pubsub/rabbitmq/metadata_test.go |
Tests metadata parsing and validation. |
pubsub/rabbitmq/rabbitmq_integration_test.go |
Adds broker integration coverage. |
tests/certification/pubsub/rabbitmq/topology_test.go |
Certifies passive topology and partitioning. |
tests/certification/pubsub/rabbitmq/enabled_plugins |
Enables the consistent-hash plugin. |
tests/certification/pubsub/rabbitmq/docker-compose.yml |
Mounts the plugin configuration. |
tests/certification/pubsub/rabbitmq/components/passive/rabbitmq-passive.yaml |
Configures passive topology certification. |
tests/certification/pubsub/rabbitmq/components/hash0/rabbitmq-hash0.yaml |
Configures the first hash consumer. |
tests/certification/pubsub/rabbitmq/components/hash1/rabbitmq-hash1.yaml |
Configures the second hash consumer. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… handling - Validate the exact routing key that will be bound; trimming during validation let " 10 " pass and still reach the broker as a non-integer binding key. - Carry the terminal subscription error through the ack channel and log it, so Subscribe reports why it gave up instead of only "not retriable". - Decorate queue PRECONDITION_FAILED with the queue name and queueDeclareMode, as exchange declares already were. - Skip queue-argument construction and validation entirely under queueDeclareMode: passive, so an externally managed queue of a type this component cannot declare is still consumable. - Validate exchangeKind against the same allowlist in both declare modes, so the runtime matches the allowedValues in metadata.yaml, which the component metadata schema treats as a hard allowlist. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Passive dead-letter exchanges are skipped, and multiple hash weights are accepted despite only one being effective.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 2
- Review effort level: Balanced
…ights - Mark passive NOT_FOUND and PRECONDITION_FAILED as terminal. Retrying them could not fix them, and retrying was actively harmful: a failed declare is a channel exception, so the retry redialled and reset() dropped the connection shared by every other subscription and publisher on the component. Subscribe now fails fast with the actionable message instead of timing out after a minute with a generic one. - Treat any *amqp.Error as a dead channel in mustReconnect, so the loop does not spend an attempt on a channel the broker has already closed. - Stop closing ackCh. subscribeForever sends at most once into a buffered channel, and closing it raced that send once the timeout fired, which panics the sidecar. Passive mode made that window routine. - Move dead lettering under queueDeclareMode. The DLX and DLQ are named from consumerID and topic at runtime, so no external topology owner can pre-create them; they belong with the queue, not with the topic exchange. Under queueDeclareMode: passive they are left entirely to the owner. - Require exactly one routingKey for a consistent hash exchange, which has one effective binding per queue, so a list silently left one weight in force. - Restore the passive-mode exchange kind leniency. Nothing at runtime enforces allowedValues; it is read only by .build-tools, so the allowlist described what can be declared and blocked plugin kinds for no benefit. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
Code reviewOne observation, non-blocking.
components-contrib/pubsub/rabbitmq/rabbitmq.go Lines 717 to 724 in 5171bf7
components-contrib/pubsub/rabbitmq/rabbitmq.go Lines 315 to 321 in 5171bf7
Nothing else came up. Reviewed with Claude Code. |
|
Filed the publish-path observation as #4596 so it is tracked rather than left in this thread — thanks @nelson-parente, the analysis there is yours. Kept out of this PR deliberately: the |
…nge-topology Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com> # Conflicts: # pubsub/rabbitmq/rabbitmq_test.go
Head branch was pushed to by a user without write access
4b20202
…ring Two scenarios nothing covered, both verified against a live broker: - A subscription to a missing externally managed exchange is a channel exception, so the shared channel dies and everyone on it reconnects once. What must not happen is that cycle repeating, which is what the terminal classification prevents; the test asserts connectionCount settles and the healthy subscription keeps delivering. - Passive topic exchange with enableDeadLetter: the component still declares the dead letter objects, whose names are only known at runtime. Signed-off-by: MyMirelHub <15373565+MyMirelHub@users.noreply.github.com>
|
Re-tested against a live RabbitMQ 4.3.6 with the consistent-hash plugin: certification suite (6 tests) and integration suite (37 tests, Added in afde5c6: passive exchange + Correction to my earlier reply on the 404 thread: the shared connection is torn down once, since a 404 is a channel exception. The terminal classification stops that repeating, not the single reconnect — |
Description
Adds two metadata options to the RabbitMQ pub/sub component, and support for the
x-consistent-hashexchange kind.exchangeDeclareModedeclare(default) /passivepassiveasserts the topic exchange already exists; never creates or modifies it.queueDeclareModedeclare(default) /passivepassiveasserts the consumer queue already exists, and leavesqueue.bindand dead-lettering to its external owner.The two are independent rather than one "external topology" switch: an operator-owned exchange paired with queues the component still declares is the combination consistent hash scale-out needs, since consumer queue names are only known at runtime.
Dead-lettering follows
queueDeclareMode, notexchangeDeclareMode. The DLX and DLQ are named fromconsumerIDand topic at runtime, so no external owner could pre-create them; they are the component's own objects and belong with the queue. UnderqueueDeclareMode: passivethey are left entirely to the queue's owner andenableDeadLetterhas no effect.Also here:
x-consistent-hashaccepted as anexchangeKind; inpassivemode any broker-supported kind is accepted, since the component never declares it.PRECONDITION_FAILEDand passive-declare-on-missing-object are rewritten into messages that name what conflicted and which option resolves it. Every other AMQP error passes through untouched.No behaviour change by default: both modes default to
declare, and existingexchangeKindvalues are unaffected.Issue reference
Closes #4553
Closes #4554
Checklist
metadata.yaml) updateddapr/docsPR: Document RabbitMQ pub/sub externally managed topology and consistent hash exchange docs#5297Tests
Unit tests cover declare/passive dispatch for exchanges and queues, the decorated and pass-through error paths, bucket-weight validation, and metadata parsing for both new fields.
Two certification tests, run through the runtime against a real broker. The suite's
docker-compose.ymlnow enablesrabbitmq_consistent_hash_exchangevia a mountedenabled_pluginsfile.TestRabbitMQPassiveTopologydeclares the exchange, queue and binding out-of-band, runs a fully passive component against them, then asserts via the management API that the broker's set of exchanges and queues is unchanged — no<consumerID>-<topic>queue, no dead-letter objects, no second exchange.TestRabbitMQConsistentHashPartitioningruns two independently scaled consumers against an operator-ownedx-consistent-hashexchange and asserts every key is handled by exactly one consumer, the key space is spread across both, and per-key order holds. A fanout-like or unordered regression fails it.