2
$\begingroup$

I'm sorry, if I ask my question by the wrong community. What is a difference between Spark ML and Flink ML and between Spark and Flink in general? The both projects are the projects of Apache, I would like to know why Foundation has two similar projects.

PS I have found some interesting article Fast Big Data: Apache Flink vs Apache Spark for Streaming Data It has answers on my question.

PPS This question has been already discussed by Stack Overflow community. The topic is What is the difference between Apache Spark and Apache Flink

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

Both Spark and Flink are designed to process data in batch or stream over distributed environment.

  • Flink primarily being defined as its ability to process streaming data in real time and being considered as good option for processing low data latency data and high fault tolerance on distributed systems on a large scale.

  • However, Spark is utilized predominantly in the case of batch processing over distributed environment.

  • Spark do have streaming API on the other hand but Flink is getting more traction for executing this need.

  • Flink can be utilized on local JVM, standalone, Yarn and cloud similarly Spark can be deployed over local, standalone, Yarn, Mesos.

There are many more difference between these two. I found some really interesting answers for your question on Quora, have a look-

Hope it helps!

Improve this answer
$\endgroup$
6
  • $\begingroup$ Glad, I could help man. Meanwhile, I'll suggest you try exploring GitHub. You can get some best practical work on these there and trust me that will be very useful. Cheers! $\endgroup$
    – Abhishek
    Oct 18, 2016 at 6:50
  • 2
    $\begingroup$ "Spark is not capable of handling data sets larger than the RAM before version 1.5.x, on the other hand Flink is." -> definitely false $\endgroup$
    – Sean Owen
    Oct 18, 2016 at 10:14
  • $\begingroup$ @SeanOwen Kindly, correct me if I'm wrong. It will help. Cheers! :) $\endgroup$
    – Abhishek
    Oct 18, 2016 at 11:11
  • 2
    $\begingroup$ Spark has never required data to fit in memory. It could process everything off disk if desired. $\endgroup$
    – Sean Owen
    Oct 18, 2016 at 12:31
  • $\begingroup$ Okay, thanks for the update. will edit the answer. Thanks! $\endgroup$
    – Abhishek
    Oct 18, 2016 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.