I found that Apache-Storm, Apache-Spark, Apache-Flink and TIBCO StreamBase are some powerful frameworks for stream processing. but I don't know which one of them has the best performance and capabilities.
I know Apache-Spark and Apache-Flink are two general purpose frameworks that support stream processing. but Apache-Storm and TIBCO StreamBase are built for stream processing specially. Is there any considerable advantage between these frameworks?
Thanks
It really depends on what you are looking to do. I love Apache Spark, but Storm has some history. I am sure as the streaming capability in Spark is built out that it will become a competitive solution. However, until Spark has some heavy hitting users (for streaming) there will remain unknown bugs.
You can also consider the community. Spark has a great community. I am not sure the level of the Storm community as I am usually the one receiving the data not handling the ingest. I can say we have used Storm on projects and I have been impressed with the real-time analysis and volumes of streaming data.
You might also want to take a look at Apache Flink. It is a new contender to Apache Spark as it can do both, realtime and batch analysis. It also claims to be faster than Apache Spark, but I think mostly in depends on your usecase.
Good thing is: Apache Flink project has some Storm compatibility layer in its development version. In this way you can take your existing Storm project and simply run it inside Flink. For me it was at least 2 times faster than Storm in a very simple topology. So give it a try! :)
Apache Storm and Apache Spark are more popular than the other ones, there are already many discussions on Quora(Storm vs Spark, Use cases for comparison).
Personally, I think Spark is a better choice.
Amazon Kinesis might be another choice for stream processing, if you don't want to set up the clusters by yourself.
Both Storm and Spark are great tools. It depends on your use-case.
I haven't tried Flink, but it looks more similar to Spark. Spark has general concept of RDD (Resilient Distributed Datasets) that can be used also for graphs, huge matrices etc.
If you want to write a word count, you any of them. But who wants a word count (except from hello world tutorials)?
Just to throw an extra option in, Microsoft Azure's Stream Analytics.
As far as a comparison of the different technologies and their relative performance, those stats seem difficult to get, but there is a useful comparison of a few technologies in this article, though doesn't mention Azure Stream Analytics. I think the difficulty is that the architectures are so different, that is makes simple metrics hard to produce. What is needed is some more robust measurement tools that can be applied to give a few metrics for various operations.
I would think that Gartner would have something to say, but the nearest I could find was their BI and Analytics quadrant, which only obliquely address stream analytics.
There are a lot of choices based on what you are looking to do.
How do you want to write the logic?
If you are looking to write code for your logic ( and not looking for a SQL like declarative language) then you can use one of Stream Processing Engines: Apache Storm, Apache Spark, Apache Fink, Apache Samza or Kafka Streams. The good news is all of them are open source.
Among them, my bet is Fink. It has a pretty nice architecture which handles many pitfalls others stepped into. Especially, if you want to do exactly once processing, Fink is the best bet.
However, writing java code to do this is fine if you do simple counting, but a nightmare if you want to do serious things like time windows, join and temporal event sequence patterns etc. ( see Patterns for Streaming Realtime Analytics for details on this argument). If you want to express the logic using a declarative language, then you will want to use Complex Event Processing, which let you write SQL-like queries and do complex things. Specifically, these kinds of queries support temporal operators like sliding windows, joins, temporal sequence patterns etc.
If you are looking for an Open source solution, the among the choices are WSO2 DAS (Data Analytics Server) formally known as WSO2 CEP ( Apache License) and Esper (GPL). ( Disclaimer I work for WSO2).
If you are OK to pay, then there is Tibco Stream Base, IBM Infosphere Streams, and SQLStreams. Both WSO2 DAS and Esper have commercial support if you need it. Paul Vincent is maintaining CEP market Survey which provides historical overview of almost all CEP engines.
Recent Forrester report surveyed the choices and you can find the details from 15 "True" Streaming Analytics Platforms For Real-Time Everything.
Above stream processing engines are adding support for SQL like query languages. However, they still lag behind with the features ( e.g. temporal sequence patterns). As I discussed in https://softwareengineeringdaily.com/2016/02/04/stream-processing-vs-complex-event-processing/, CEP and Stream processing engines are merging.
Scalability
If you plan to process millions of events per second, then you need to setup a complex distributed processing graph. On this topic, stream processing has an advantage where they are historically built to scale. However, as I mentioned, these two models are merging.
IoT Analytics
If you are doing an IoT analytics, then the data creates a time series. In that case, often you need to do temporal operations such as sliding windows. On that use case, you are better off with an SQL like language hence I recommend to go for CEP.
Streaming + Batch
The final twist is whether you want to do batch ( MapReduce style processing) as well. If yes, read about Lambda Architecture and Kappa architecture. Apache Fink and WSO2 DAS supports this scenario. ( and there may be others).
Flink is an upcoming good equivalent to Spark (or even better) as it process each tuples unlike micro batch concept of spark streaming.
