1
$\begingroup$

Basically what the title is. The problem I currently have is that my dataset consists of 2.8 billion rows, and I have it as a Pyspark data frame.

I want to use some library such as FLAML for finding hyperparameters, but I cannot do it with a Pyspark data frame.

The solution I’ve thought of currently is:

  • Keep the overall distribution of the data and scale it down to 60 or 70 million rows so it’s usable as a Pandas data frame
  • Use FLAML to find hyperparameters for this dataset on some model, say XGBoost

Now my question is if these hyperparameters will work if I trained the model with 2.8 billion rows and the same hyperparameters I found from tuning.

It would be great if someone can additionally point to any research done in this regard.

$\endgroup$

1 Answer 1

1
$\begingroup$

You could use a subsample to select the hyperparameters, and test in the rest of your data. And make sure to make this experiment several times to assess the stability of the hyperparameter tuning algorithm.

And another thing that is possible to point is using only a simple linear model and assess the performance with a train-test split, with this massive data set a simple validation is likely to work, and see if meet your requirements. Another interesting line of research is use more robust big data tools such as pure Spark in Scala with more ram. https://spark.apache.org/docs/latest/ml-tuning.html

$\endgroup$
1
  • $\begingroup$ The problem here is choosing a paramgrid though. In my experience FLAML works very well in finding hyperparameters by using some iterative process. If I’m using XGBoost for example, I have no idea what is the possible feature space for some feature like number of estimators for 2.8 billion rows. $\endgroup$ Jul 19, 2022 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.