Timeline for Sklearn Pipeline for mixed features: numerical and (skewed) categorical
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 21, 2020 at 14:50 | comment | added | Sahil Gupta | I explored my problem a bit more and realized that although LASSO is ideally a feature selection technique, due to the skewed counts it becomes much harder to even train a good LASSO model. So, to even reach a viable model for feature selection, it seems like a good idea to reduce the number of categories. Aggregating the rare categories into a new group (called 'other') seems like a good idea. | |
| Mar 21, 2020 at 14:46 | vote | accept | Sahil Gupta | ||
| Mar 19, 2020 at 17:27 | comment | added | Sahil Gupta | A continuation of my previous comment. But, even in such a case, I guess the probability of this happened would be very low considering the validation set (and the train set) will be randomly sampled. I wonder if the developers already thought of this edge case while designed the one-hot encoder. | |
| Mar 19, 2020 at 17:16 | comment | added | Sahil Gupta | The rationale behind 'ignore' for test data makes sense because that's something for which literally, we have no information. Sure, you may be right that coefficient for a rare type (eg. Metal) may result in overfitting, but that would be taken care of by the regularization term. I mean that's the whole point of using a LASSO model here. I guess my question is geared more towards the design of API. You can surely come up with examples similar to above where the counts are relatively less skewed and depending on the size of validation set you end up in a similar situation. | |
| Mar 19, 2020 at 2:59 | history | answered | Ben Reiniger♦ | CC BY-SA 4.0 |