2
$\begingroup$

I'm sorry, if I ask my question by the wrong community. What is a difference between Spark ML and Flink ML and between Spark and Flink in general? The both projects are the projects of Apache, I would like to know why Foundation has two similar projects.

PS I have found some interesting article Fast Big Data: Apache Flink vs Apache Spark for Streaming Data It has answers on my question.

PPS This question has been already discussed by Stack Overflow community. The topic is What is the difference between Apache Spark and Apache Flink

$\endgroup$
4
$\begingroup$

Both Spark and Flink are designed to process data in batch or stream over distributed environment.

  • Flink primarily being defined as its ability to process streaming data in real time and being considered as good option for processing low data latency data and high fault tolerance on distributed systems on a large scale.

  • However, Spark is utilized predominantly in the case of batch processing over distributed environment.

  • Spark do have streaming API on the other hand but Flink is getting more traction for executing this need.

  • Flink can be utilized on local JVM, standalone, Yarn and cloud similarly Spark can be deployed over local, standalone, Yarn, Mesos.

There are many more difference between these two. I found some really interesting answers for your question on Quora, have a look-

Hope it helps!

$\endgroup$
6
  • $\begingroup$ Glad, I could help man. Meanwhile, I'll suggest you try exploring GitHub. You can get some best practical work on these there and trust me that will be very useful. Cheers! $\endgroup$ Oct 18 '16 at 6:50
  • 2
    $\begingroup$ "Spark is not capable of handling data sets larger than the RAM before version 1.5.x, on the other hand Flink is." -> definitely false $\endgroup$
    – Sean Owen
    Oct 18 '16 at 10:14
  • $\begingroup$ @SeanOwen Kindly, correct me if I'm wrong. It will help. Cheers! :) $\endgroup$ Oct 18 '16 at 11:11
  • 2
    $\begingroup$ Spark has never required data to fit in memory. It could process everything off disk if desired. $\endgroup$
    – Sean Owen
    Oct 18 '16 at 12:31
  • $\begingroup$ Okay, thanks for the update. will edit the answer. Thanks! $\endgroup$ Oct 18 '16 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.