6
$\begingroup$

I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.

https://spark.apache.org/docs/2.2.0/mllib-feature-extraction.html#word2vec https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec

mllib seems to be for using RDD's. And ml seems to be using "DataFrames".

What is the difference? Shouldn't they both be using RDDs if this is spark under the hood?

What is a "DataFrame" here? As the documentation doesn't explain it.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

You are right, mllib uses RDDs and ml uses dataframes. At the beginning, there was only mllib because dataframes did not exist in spark. In fact, ml is kind of the new mllib, if you are new to spark, you should work with ml and dataframes.

Improve this answer
$\endgroup$
5
$\begingroup$

An RDD is a Read-only partition collection of records. RDD is or was the fundamental data structure of Spark. It allows a programmer to perform in-memory computations on large clusters in a fault-tolerant manner. Thus, speed up the task.

A dataframe on the other hand organizes data into named columns. It is an immutable distributed collection of data. DataFrame in Spark allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction.

The two are very different and more details on the two can be found on this link https://data-flair.training/blogs/apache-spark-rdd-vs-dataframe-vs-dataset/

mllib was in the initial releases of spark as at that time spark was only working with RDDs. Now a lot of Spark coding is done around dataframes, which ml supports. Spark has also put mllib under maintenance.

As of Spark 2.0, the RDD-based APIs in the spark.mllib package have entered maintenance mode. The primary Machine Learning API for Spark is now the DataFrame-based API in the spark.ml package.

Q. What are the implications?

MLlib will still support the RDD-based API in spark.mllib with bug fixes. MLlib will not add new features to the RDD-based API. In the Spark 2.x releases, MLlib will add features to the DataFrames-based API to reach feature parity with the RDD-based API. After reaching feature parity (roughly estimated for Spark 2.3), the RDD-based API will be deprecated. The RDD-based API is expected to be removed in Spark 3.0.

Q. Why is MLlib switching to the DataFrame-based API?

DataFrames provide a more user-friendly API than RDDs. The many benefits of DataFrames include Spark Datasources, SQL/DataFrame queries, Tungsten and Catalyst optimizations, and uniform APIs across languages. The DataFrame-based API for MLlib provides a uniform API across ML algorithms and across multiple languages. DataFrames facilitate practical ML Pipelines, particularly feature transformations. See the Pipelines guide for details.

Q. What is “Spark ML”?

“Spark ML” is not an official name but occasionally used to refer to the MLlib DataFrame-based API. This is majorly due to the org.apache.spark.ml Scala package name used by the DataFrame-based API, and the “Spark ML Pipelines” term we used initially to emphasize the pipeline concept.

Q. Is MLlib deprecated?

No. MLlib includes both the RDD-based API and the DataFrame-based API. The RDD-based API is now in maintenance mode. But neither API is deprecated, nor MLlib as a whole.

You will find more details on Sparks official release https://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.