Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am working with a dataset where users have a set of skills. I have more than 500 skills and I was wondering what is the best way of encoding a vector, e.g., ['java', 'python', 'c'] into a feature, so it would be possible to use the user skills as features.
I thought about one-hot-encoding but I am afraid of the curse of dimensionality, since we have hundreds of skills.
Any suggestion on how to handle this kind of scenarios?
I think "one hot" is the obvious thing to do. 500 features is usually not a problem (if you don‘t have too few observations). In any case you could look into "shrinking" features by using Lasso/Ridge for instance.
Probably you could also look into dimensionality reduction, e.g. by using principle components (PCA).
You could also do some feature selection in the sense that you "kick out" features (skills) which to not have much predictive power or which are redundant. You could for instance check for (very) high correlation among skills or remove skills with little importance after fitting some random forest.
