I have very high dimensional data. Almost 20% of the columns has different value in less than 1% of rows. All of these are binary columns and many columns has 0s filled in more than almost 98% of rows.

Some more info: Target variable is an imbalanced(91.9%:8.1%) binary variable.
Every variable I have, except 3, are binary.

I would like some ideas on how to deal with columns like this? drop them or smote to have more data?

Thanks in advance.

  • 1
    $\begingroup$ What did you try so far? With tree based methods, such as boosting, it should be no big problem to include these columns, but they will probably contribute little. One problem may be (high) correlation between columns. This could be an issue when you use logit-like methods, such as Lasso. $\endgroup$
    – Peter
    Aug 30 '20 at 9:55
  • 1
    $\begingroup$ with these column included, light gbm is giving good results so far. I am about to try excluding those columns and see how the results vary. surprisingly, fixing the skew and even log transformation of 3 continuous columns i have didn't change auc by even 0.000001. that's all I have tried till now @Peter $\endgroup$ Aug 30 '20 at 10:13
  • 2
    $\begingroup$ Sounds okay. Transforming cols often comes with little effect in tree-based methods. I guess lightGBM is a good choice, probably with a good deal of regularization $\endgroup$
    – Peter
    Aug 30 '20 at 10:17
  • 1
    $\begingroup$ You tagged your question with "anomaly-detection", but you say you have a some target values. Are you trying to detect the class, or anomalies? $\endgroup$ Aug 30 '20 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.