0
$\begingroup$

I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.

https://spark.apache.org/docs/2.2.0/mllib-feature-extraction.html#word2vec https://spark.apache.org/docs/2.2.0/ml-features.html#word2vec

mllib seems to be for using RDD's. And ml seems to be using "DataFrames".

What is the difference? Shouldn't they both be using RDDs if this is spark under the hood?

What is a "DataFrame" here? As the documentation doesn't explain it.

$\endgroup$
0
$\begingroup$

You are right, mllib uses RDDs and ml uses dataframes. At the beginning, there was only mllib because dataframes did not exist in spark. In fact, ml is kind of the new mllib, if you are new to spark, you should work with ml and dataframes.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.