Repository navigation
Beyond the pipeline - DAG / Flow Graph / Data Flow #6
Description
Is it possible to add support to create flows beyond a pipeline by being able to create a flow graph like in:
http://akka.io/news/2014/09/12/akka-streams-0.7-released.html
Activity
palladin commentedon Dec 16, 2014
Thank you for your suggestion! Actually, I'm thinking of possible ways to combine the power of Streams with other libraries like Rx and TPL Dataflow. The API of Akka Streams looks very interesting and a good source for inspiration.
sirinath commentedon Dec 16, 2014
Also have a look at: https://github.com/Hopac/Hopac
Main thing is ease of use. If you have a DSL like Akka this would be easy to use.
Also see:
http://www.nextflow.io/ (DataFlow)
http://noflojs.org/ (Flow Based Programming)
And to a lesser extent:
http://www.infoq.com/presentations/mantis (Rx, DAG, Functional Reactive Programming)
FBP:
http://nifi.incubator.apache.org/
DAG:
http://flink.incubator.apache.org/
palladin commentedon Dec 16, 2014
Thanks for all the wonderful resources!
sirinath commentedon Dec 16, 2014
Integration with Rx and TPL is very welcome and good. But I am not sure if they are the best performing options. Have a look at Hopac for better performance.
palladin commentedon Dec 16, 2014
I love Vesa's work and Hopac is definitely on my radar!
sirinath commentedon Dec 16, 2014
Great
sirinath commentedon Dec 25, 2014
The docs have come out. http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html?_ga=1.257897355.1234628154.1418042628
It wold be great if you can have a similar DSL.
palladin commentedon Dec 25, 2014
Thanks for the info. I really like the (~>) as the F# idiomatic operator for the underlying TPL.Dataflow LinkTo method.
sirinath commentedon Dec 26, 2014
Yes. This would be good for starters.
But perhaps may be explore if there are better performing option that TPL also.
sirinath commentedon Dec 26, 2014
Design it so you mix and match the underling framework.
[-]Beyond the pipeline - DAG / Flow Graph[/-][+]Beyond the pipeline - DAG / Flow Graph / Data Flow[/+]sirinath commentedon Mar 9, 2015
Wondering how things are shaping up in the data flow front?
palladin commentedon Mar 9, 2015
Currently my focus is on MBrace.Streams (CloudStreams) but the data flow integration is a big concern for us. Unfortunately, the local-streams data flow support must wait for the Cloud data flow plans to take shape.
sirinath commentedon Mar 9, 2015
I am more interested in Cloud data flows. If this is taking time at least the local case.