🌟 Roadmap 🌟

[Lancetnik]

Roadmap

We know that some users are waiting for new broker support (like SQS/MQTT/etc.), but please be patient a bit longer.

Due to the inner project structure becoming much harder to change with each new broker, we are now focusing on stabilizing the inner and outer API. Then we will be able to promote new broker support in an easier way (like we used to do before).

Road to 1.0.0

Polish Current Features

First of all, we should polish already existing brokers and core features:

• Complete Observability features
  • Add baggage feature support to tracing (assigned to @draincoder, Add OTel baggage support #1692)
  • Correct spans linking in batches case (assigned to @draincoder, fix: correct spans linking in batches case #1532)
  • Prometheus Middleware to collect metrics (Feature: Prometheus Middleware to collect metrics #1665)
  • Healthcheck mechanism for k8s-probes (Feature: add K8S probes checker API #1181)
  • Complete observability documentation
• Strong Pseudo-random values generation (like message IDs)
  • Add Nuid or another pseudo-random algorithm implementation (Feature: more stronger unique RPC subject names #1386)
  • Pass this object throughout the whole application to all places generating random values (Feature: correlation_id factory function at broker creation time, e.g. for ulids #2570)
• Extended RMQ features (feat: extented RMQ features #1522)
  • Connections and channels pool support (Feature: use aio_pika.Pool to connect RMQ #975)
  • Allow using a new channel for specific requests
  • Concurent RPC requests (using channel pool) (Feature: Allow concurrent RPC calls on rabbit implementation #1318)
  • Publish Transactions support
• Extended NATS features
  • Suitable way to subscribe on multiple subjects in JetStream (Feature: support NATS filter subscription #1518)
  • Create KeyValue adapter to provide custom methods
  • Create ObjectStore adapter to provide custom methods
  • In-memory KeyValue and ObjectStore tests
• Extended Kafka features
  • Kafka manual commit support (Bug: Event ack'd after calling KafkaMessage.nack with auto_commit=False #1001)
  • Transactions support (Feature: add Kafka transactions support #1157)
  • Pattern subscription
• Extended Redis support
  • Add Redis Cluster support (Feature: Adds support for the Redis cluster #1170)
  • Concurrent messages processing (feature: concurrent Redis consuming #1507)

Inner Structure Refactoring

The next step is making Context non-global: we should pass it throughout the whole application from FastStream object -> Broker -> Subscriber -> Call. This step is also related to FastDepends 3.0 migration, and we can refactor dependency_overrides as well, and finally make Pydantic replaceable in serialization cases.

• Optional Pydantic (0.6.0 #1779)
  • Migrate to FastDepends 3.0 (0.6.0 #1779)
  • Provide an option to use custom Serializer Protocol implementation (0.6.0 #1779)
• Local dependency-overrides (0.6.0 #1779)
  • Migrate to FastDepends 3.0 (0.6.0 #1779)
  • Pass dependecy-overrides container from FastStream to the final consumers (0.6.0 #1779)
  • Complete Depends documentation by overrides section
• Local context (0.6.0 #1779)
  • Refactor all parsers to get regex in init instead of using context
  • Refactor ContextType to use local context (0.6.0 #1779)
  • Pass context to FastDepends as kwargs (0.6.0 #1779)
  • Pass context from FastStream to the final consumers (0.6.0 #1779)
  • Refactor logging to not use raw context (0.6.0 #1779)
• Finally, after all these steps, we can finalize the Middleware Public API (0.6.0 #1779)
  • Use Response DTO instead of options (0.6.0 #1779)
  • Add local context as an init option (0.6.0 #1779)

New in Testing

We plan to refactor the In-Memory TestClient to make it side-effect-less (currently, it has some problems with reusing).

• New TestClient logic (Bug: Adding a subscriber to a TestRabbitBroker persists between test runs #1036, Bug: starting the broker breaks in-memory behaviour #1479)
• Add tests for context (Support checking keys in tests #557)

New Brokers Support

Finally, we can take a break from refactoring and add new broker support:

• SQS support (SQS support #794)
• MQTT support (MQTT support #956)

1.0.0 Finalization

At the end of the year (hopefully), we plan to release the first major version - 1.0.0

• Public Multi-broker support (Using multiple brokers #526)
  • AsyncAPI schema merging
  • Cross-brokers In-Memory Tests
  • Correct logging resolution
• Extended subscriber API
  • broker.subsciber("test").get_one() - get concrete messages support (Add broker.subscriber.get_one() #1726)
  • async for msg in broker.subscriber("test") - messages iterator support (Feature: subscriber iteration support #1881)

Community

Additionally, we are planning to start our Experts activity and provide users with a roadmap to study FastStream, a form to approve your expertise, and a public page with all approved experts to encourage people to help each other (and boost their resumes with the FastStream expert mark).

• Launch Experts initiative

AsyncAPI

Last but not least, we should migrate to the new major AsyncAPI version

• AsyncAPI 3.0 support (AsyncAPI 3.0 support #980, 0.6.0 #1779)
• AsyncAPI 2.6 compatibility (0.6.0 #1779)

1.0.0 +

We are not looking too far ahead, but we have few thoughts:

• RabbitMQ Streams binary protocol support
• Generate FastStream project from AsyncAPI specification
• Enable users to publish a test message from AsyncAPI Web right to the app (Feat: React children components support asyncapi/asyncapi-react#840)
  • With In-memory execution mode
  • With real broker publishing

Eventhough brokers are good, brokerless is much better, so ZMQ support looks promising to us

• ZMQ support (ZMQ support #1142)

And the last thing - we want to provide you with some persistence layer (like Faust table or smth) to allow windowing incoming messages stream.

• Incoming stream windowing

P.S. This plan is still pretty raw, and we can swap any of these points, but this is the general direction.

[r0kk]

Hi thank you for sharing the roadmap, I would like to point out that community can already use MQTT protocol since RabbitMQ broker supports MQTT protocol via plugin. This means that FastStream can be used for with MQTT protocol. Maybe tutorial or announcement could inform community about this option.

Thank you for all contributions.

[Lancetnik]

@r0kk thank you for your kind words! But I am not sure, that plugin is full-MQTT compatible. Is it suitable for all MQTT cases? Did you test FastStream with it?
Anyway, we plan to implement MQTT support right after 0.6.0 release (it should be in a few months), it's not too long to wait

[matheusvnm]

@Lancetnik I just came across this project and found out that I'm currently doing something similiar for my company. We currently only work with PubSub via Google Platform with New Relic. @Lancetnik, do your team has the intention to provide support to Google PubSub?

If so, I have a similar implementation for this "broker". In case you see some benefit in adding support for Google PubSub (and maybe New Relic optional instrumentation) I would like to contribute to the project.

[Lancetnik]

@matheusvnm, thank you for your interest and offer of help!

We are planning to introduce new broker support, and GCP is also on the plan - #1229.
However, my current priorities are MQTT and SQS.

Fortunately, we have an active contributor working on GCP support - #2444 . It seems like he has already done a lot of work, but I literally know nothing about GCP and am unable to deeply validate his work.

It would be great if you could coordinate and complete the task together - two heads (and pairs of hands) are much better than one. What do you think of this opportunity?

[rsJames-ttrpg]

@matheusvnm Hi, I am working on the gcp broker atm slowly progressing it. I could definitely do with a second pair of eyes if you want contribute. I don't have a huge amount of time at the moment but in a few weeks I should be able to dedicate actual time to it. In the mean time if you want to take a look at the pr and make sure it covers your use case / any missing gcp specific features, I have worked pretty extensively with pubsub but there are some features that I haven't touched and there are some of the gcp client features I have deliberately not implemented. If you want to chat I am on discord

[matheusvnm]

Hey, thanks for reaching out @rsJames-ttrpg! I'm definitely happy to contribute and collaborate on this.

I'm currently looking at the overall project to understand the main concepts and how it is structured. Once I do that I will
take a look at the PR to get familiar with your current progress. I can certainly check it against my use cases and provide feedback on any GCP-specific features that might be missing.

No worries about your time constraints; I too have a very time constrained schedule. I'll connect with you on Discord to chat more about the project on the weekend!

Thank you!