I have a problem related to clustering, where i need to cluster skill set from job domain.

Let's say, in a resume a candidate can mention they familiarity with amazon s3 bucket. But each people can mention it in any way. For example,

  1. amazon s3
  2. s3
  3. aws s3

For a human, we can easily understand these three are exactly equivalent. I can't use kmeans type of clustering because it can fail in a lot of cases.

For example,

  1. spring
  2. spring framework
  3. Spring MVC
  4. Spring Boot

These may fall in the same cluster which is wrong. A candidate who knows spring framework might not know sprint boot etc.,

The similarity of a word based on the embeddings/bow model fails here.

What are the options I have? Currently, I manually collected a lot of word variations in a dict format, key is root word value is an array of variations of that root word.

Any help is really appreciated.

  • $\begingroup$ This is potentially a good use case for topic modelling such as LDA $\endgroup$ Aug 21, 2021 at 21:28
  • 1
    $\begingroup$ @PirateofeBay What type of data do we need for such, that's the problem I am facing? There's no opensource dataset I could find. I have a huge set of skills alone but without context, there's no use. Please do provide any ideas on how to get started on this w.r.t data and modeling $\endgroup$
    – Sai Kumar
    Aug 21, 2021 at 21:33

1 Answer 1


That is commonly called entity linking, the task of assigning a unique identity to entities. Your issue in particular is name variations, the same entity might appear with different textual representations / surface forms.

Clustering is not the most useful way of solving name variations since clustering is unsupervised.

There are many ways to approach resolving name variations. Given that job skills is a relatively common domain, you can find or pay for existing mapping of job skill entities. If you want to build your own system, most systems start with hand-coded rules (typically a combination of regular expressions and hash maps). After diminishing returns with a hand-coded rules, other models could be applied. A knowledge base can be used to disambiguate textual entities. Again, since job skills is a common domain there are many existing knowledge bases. You could create your own job skill knowledge base but that would be a complex, slow, and error-prone process.

  • $\begingroup$ "there are many existing knowledge bases", can you explain what you mean by creating a knowledge base? Are there data already available for building a knowledge base I tried looking couldn't find anything relevant. Also, how can one start by creating knowledge base, can you provide some technical explanation. It's fine a complex, slow and error-prone process I think is a good step in the right direction. $\endgroup$
    – Sai Kumar
    Aug 28, 2021 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.