0
$\begingroup$

I am using spark/scala to do some transformations on a spark dataframe. I have the following code:

val predictionAndLabelsRDD = predictionAndLabels
    .select("label", "predictions")
    .map([REFERENCE TO INTERMEDIARY DATAFRAME].columns.map(col(_).cast("Int")) : _*)
    .rdd

In the code above, I want to reference the spark dataframe that is created after the select operation. I know I can do the following:

val temporaryDF = predictionAndLabels
    .select("label", "predictions")

val predictionAndLabelsRDD = temporaryDF
    .select(temporaryDF.columns.map(col(_).cast("Int")) : _*)
    .rdd

But the first example seems more elegant. Any ideas on how to actually do this?

Improve this question
$\endgroup$

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.