Join two dataframes - Spark Mllib

Question

I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.

How can I return only the details of the student that have positive grade (make the join) but not using SQL Context.

I've this code:

val all_students = sc.textFile("/user/cloudera/Data");

case class Students(Customer_ID:String,Name:String,Age:String);

def MyClass(line: String) = {
val split = line.split(',');
Students(split(0),split(1),split(2))
}

val df = all_students.map(MyClass).toDF("Customer_ID","Name", "Age").select("Customer_ID","Name", "Age");

val students_positive_grande = sc.textFile("/user/cloudera/Data");

How can I make the join between this datasets? I want to join the "Customer_ID" with the first column of the second column...

Answer 1

Use this syntax:

val joinedDF = students_positive_grande.as('a).join(
                          df.as('b),
                          $"a.Customer_ID" === $"b.Customer_ID")

joinedDF.select($"a.Customer_ID", $"b.Customer_ID")