1
$\begingroup$

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.

Since the datasets are so big (millions of rows) when I used StringIndexer in pyspark to get dummies from first names, I got the following error:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 25.0 failed 4 times, most recent failure: Lost task 0.3 in stage 25.0 (TID 399, 10.139.64.28, executor 2): org.apache.spark.SparkException: Failed to execute user defined function(StringIndexerModel$$Lambda$6517/699548305: (string) => double)

Can you suggest any better approach to convert the first names to dummy variables?

$\endgroup$
1
  • 1
    $\begingroup$ I don't know spark and I have no idea what this error means, but this is certainly a case where discarding the least frequent names first is necessary. $\endgroup$
    – Erwan
    Commented Nov 4, 2021 at 12:13

1 Answer 1

1
$\begingroup$

It appears that you are writing a user defined function to parse the data Failed to execute user defined function(StringIndexerModel$$Lambda$6517/699548305: (string). It is better to use the built-in Spark functions (which are more scalable).

After you parse the string, use OneHotEncoderEstimator to generate the dummy variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.