0
$\begingroup$

I have a large dataset (40 mil rows, 50 columns) with mostly categorical columns (some of them are numerical) and I am using Python/Pandas. Categorical columns have up to 3000 unique labels.

I am looking for best practices on how to approach this. Obviously one-hot encoding (OHE) as it is is out of question. I have tried to make smaller number of categories and do OHE in that way but the model was very bad, a lot of information is being lost. Also, memory is an issue and everything takes a long time.

Should I sample the data in this case? If so, how? Categorical columns depend on each other, they are nested. Label encoding and other encoders also didn't show good results. I have tried CatBoost Regressor and other tree like models. How would you approach this problem starting from data visualisation, feature engineering sampling, modelling?

$\endgroup$
6
  • 1
    $\begingroup$ Why is OHE out of the question? $\endgroup$
    – Dave
    Commented Mar 21, 2022 at 10:06
  • $\begingroup$ Because I get around 60 000 new columns after plain OHE and the memory dies. $\endgroup$
    – Chris
    Commented Mar 21, 2022 at 10:09
  • 1
    $\begingroup$ a couple of comments: 1. you can divide your dataset into batches if it does not fit in memory. 2. PCA might be a technique you wanna look at. $\endgroup$
    – Oscar
    Commented Mar 21, 2022 at 15:36
  • $\begingroup$ You can still use plain OHE with sparse matrix representation to avoid memory problems. The scikit-learn OneHotEncoder uses sparse matrices by default. However I agree with @Oscar that you should also consider dimensionality reduction techniques like PCA. Especially since you say "Categorical columns depend on each other, they are nested" $\endgroup$
    – zachdj
    Commented Mar 21, 2022 at 17:17
  • $\begingroup$ Another simpler dimensionality reduction step might be to eliminate all categorical columns except the final layer of nesting. With nested categories, the final nesting level contains all of the information in previous levels. For example, if you have categories "Animals", "Animals > Mammals", "Animals > Mammals > Cat", "Animals > Mammals > Dog", then it would be sufficient to discard everything except "Cat" and "Dog". $\endgroup$
    – zachdj
    Commented Mar 21, 2022 at 17:20

1 Answer 1

1
$\begingroup$

With such a big dataset, I would start using a random sample of the data as a smaller training set, until you have identified an algorithm that is suitable.

With so many descriptors, I would start with a Random Forest classifier. Although not necessarily the best final model, it is a good way to explore feature importance.

Happily, these choices fit together well: you can train each iteration of the forest building with a different subsample from the dataset.

$\endgroup$
1
  • $\begingroup$ Thank you, this makes sense, I will try it. $\endgroup$
    – Chris
    Commented Mar 23, 2022 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.