I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and add_column while allowing for a user-defined function doesn't seem to allow multiple return values - so does anyone have a recommendation how I would accomplish this? I'd prefer only calling the generating function d,e,f=f(a,b,c) once per row, as its expensive.

  • $\begingroup$ Have a look here for some ideas. They are generally only adding single columns, but perhaps you can make that do what you require. $\endgroup$
    – n1k31t4
    Jun 26 '18 at 10:31
  • 1
    $\begingroup$ this would seem to be pretty close tho more involved than what i was hoping for $\endgroup$ Jun 26 '18 at 13:43
  • $\begingroup$ stackoverflow.com/questions/35322764/… $\endgroup$
    – Dyno Fu
    Oct 6 '18 at 18:10
  • 1
    $\begingroup$ You can create an udf function, and return a dictionary, to this is needed define the struct of the dict, an implemented solution of this method is here $\endgroup$ Nov 29 '18 at 15:51
  • 2
    $\begingroup$ ironic, that this hit 10K views and is off-topic $\endgroup$ Jun 4 '20 at 12:55

Browse other questions tagged or ask your own question.