Computed a column using a formula (formula does't involve any log functions, just a group by with .sum()), but as expected this column would result in infinite/exponential values like below:

 7.3297+23 ...etc (similar data)

My doubt is how should I handle this column as I could also not avoid this column for prediction for a binary classification problem. What are the techniques to use before using this column in model classifier.



Depends a bit on the model you are going to run. I will explain a bit for Linear Models and for Decision trees ensembling (gradient boosting and random forest)

Decision tree

Not much to do, when the tree is built each branch will choose a split. If there is any information gain with a large value, it will choose it and make a split.

Some implementations, as catboost have a quantization in it. The split could be done eventually in something like 95% quantile and that will work with large values.

Generalized Linear Models

You will need to change this. My suggestion is creating a feature based in the values of the column (if df.col1> 9999: 1 else 0). And then Winsorizing by a certain threshold.

  • $\begingroup$ Thanks for the answer @Carlos Mougan! If I'm going to start with Logistic Regression, once I do this if df.col1> 9999: 1 else 0, that feature column would have 0's or 1's, how can I decide on the limits for winsorizing as per docs $\endgroup$
    – omdurg
    May 18 '20 at 9:35
  • $\begingroup$ It's a parameters, you can optimize parameters in Cross Validation or if you are not sure how to create a custom pipeline and crossvalidate with it, you can just try and error. There is not much theory behind this, it is just experimental. $\endgroup$ May 18 '20 at 9:38
  • $\begingroup$ Done! @Carlos Mougan, thanks! $\endgroup$
    – omdurg
    May 18 '20 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.