2
$\begingroup$

I am working with a dataset where users have a set of skills. I have more than 500 skills and I was wondering what is the best way of encoding a vector, e.g., ['java', 'python', 'c'] into a feature, so it would be possible to use the user skills as features.

I thought about one-hot-encoding but I am afraid of the curse of dimensionality, since we have hundreds of skills.

Any suggestion on how to handle this kind of scenarios?

$\endgroup$

1 Answer 1

4
$\begingroup$

I think "one hot" is the obvious thing to do. 500 features is usually not a problem (if you don‘t have too few observations). In any case you could look into "shrinking" features by using Lasso/Ridge for instance.

Probably you could also look into dimensionality reduction, e.g. by using principle components (PCA).

You could also do some feature selection in the sense that you "kick out" features (skills) which to not have much predictive power or which are redundant. You could for instance check for (very) high correlation among skills or remove skills with little importance after fitting some random forest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.