Suppose that I have a file which has thousands of skills starting from A-Z. Now, I would like to create a model that can group similar skills together (example neural network and SVM can group together). I know that I can use NLP for this problem, but I'm not sure about the algorithm that I can use to get the best result.

I'm new to NLP so any help is greatly appreciated.


If all you have is the name/word, then I see only two ways:

  1. You can look at word similarity, basically by measuring how similar the spelling is. There are a number of possible metrics (e.g. Levenshtein distance). This will probably not get you very far.
  2. You can look at semantic similarity. You can use pre-trained word embeddings to map the words to a new vector space where you can calculate the distance between the word embeddings, e.g. with word2vec or other implementations. Alternatively, you can use a dictionary/corpus (e.g. https://www.nltk.org/index.html) that provides you a word taxonomy that allows you to compute semantic similarity between words via relationships like synonyms etc. See here for different approaches.
  • $\begingroup$ Can you show me some example? $\endgroup$ – Siddhant Singh May 9 '19 at 21:05
  • $\begingroup$ what do you mean by pre-trained word embeddings? $\endgroup$ – Siddhant Singh May 11 '19 at 7:33
  • $\begingroup$ if you are unfamiliar with these concepts it's probably best to review them first. if you have a specific question about them, you can open a new question. otherwise your question is too broad to be answered in this type of forum :-) $\endgroup$ – oW_ May 12 '19 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.