I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.

How can I return only the details of the student that have positive grade (make the join) but not using SQL Context.

I've this code:

val all_students = sc.textFile("/user/cloudera/Data");

case class Students(Customer_ID:String,Name:String,Age:String);

def MyClass(line: String) = {
val split = line.split(',');

val df = all_students.map(MyClass).toDF("Customer_ID","Name", "Age").select("Customer_ID","Name", "Age");

val students_positive_grande = sc.textFile("/user/cloudera/Data");

How can I make the join between this datasets? I want to join the "Customer_ID" with the first column of the second column...


1 Answer 1


Use this syntax:

val joinedDF = students_positive_grande.as('a).join(
                          $"a.Customer_ID" === $"b.Customer_ID")

joinedDF.select($"a.Customer_ID", $"b.Customer_ID")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.