An RDD is a Read-only partition collection of records. RDD is or was the fundamental data structure of Spark. It allows a programmer to perform in-memory computations on large clusters in a fault-tolerant manner. Thus, speed up the task.
A dataframe on the other hand organizes data into named columns. It is an immutable distributed collection of data. DataFrame in Spark allows developers to impose a structure onto a distributed collection of data, allowing higher-level abstraction.
The two are very different and more details on the two can be found on this link https://data-flair.training/blogs/apache-spark-rdd-vs-dataframe-vs-dataset/
mllib was in the initial releases of spark as at that time spark was only working with RDDs. Now a lot of Spark coding is done around dataframes, which
ml supports. Spark has also put
mllib under maintenance.
As of Spark 2.0, the RDD-based APIs in the spark.mllib package have
entered maintenance mode. The primary Machine Learning API for Spark
is now the DataFrame-based API in the spark.ml package.
Q. What are the implications?
MLlib will still support the RDD-based API in spark.mllib with bug
fixes. MLlib will not add new features to the RDD-based API. In the
Spark 2.x releases, MLlib will add features to the DataFrames-based
API to reach feature parity with the RDD-based API. After reaching
feature parity (roughly estimated for Spark 2.3), the RDD-based API
will be deprecated. The RDD-based API is expected to be removed in
Q. Why is MLlib switching to the DataFrame-based API?
DataFrames provide a more user-friendly API than RDDs. The many
benefits of DataFrames include Spark Datasources, SQL/DataFrame
queries, Tungsten and Catalyst optimizations, and uniform APIs across
languages. The DataFrame-based API for MLlib provides a uniform API
across ML algorithms and across multiple languages. DataFrames
facilitate practical ML Pipelines, particularly feature
transformations. See the Pipelines guide for details.
Q. What is “Spark ML”?
“Spark ML” is not an official name but occasionally used to refer to
the MLlib DataFrame-based API. This is majorly due to the
org.apache.spark.ml Scala package name used by the DataFrame-based
API, and the “Spark ML Pipelines” term we used initially to emphasize
the pipeline concept.
Q. Is MLlib deprecated?
No. MLlib includes both the RDD-based API and the DataFrame-based API.
The RDD-based API is now in maintenance mode. But neither API is
deprecated, nor MLlib as a whole.
You will find more details on Sparks official release https://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api