1
$\begingroup$

I have a huge collection of skills collected/scraped from various online sources. It was huge effort done by our team.

Now, the biggest challenge we are facing is trying to normalize the skills back to its root form to reduce the duplication.

Here's are few examples,

agile    (or) agile methodologies  (or) agile software
java 2.x (or) java 3.x (or) java

These kind of duplicates are very common is my datasets. So I'm looking for a way to normalize them.

I am not very sure of the way to solve this issue. Can anyone suggest me some very good ways we can solve this problem with some decent accuracy?

Thank you

Improve this question
$\endgroup$
2
  • $\begingroup$ You can basically use regex to identify similar values and normalize them by assigning a common root node manually $\endgroup$
    – Anoop A Nair
    Commented Nov 13, 2020 at 19:51
  • $\begingroup$ @AnoopANair Can you explain your thought process on how you would do it for few examples? How will you determine agile and agile methodologies as one? How would you group it? $\endgroup$
    – user_12
    Commented Nov 13, 2020 at 20:10

1 Answer 1

Reset to default
1
$\begingroup$

In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a few API. Regex, Levenshtein distance, and other old school metrics are great, but in general you are never certain as to what you want to consider "as similar" besides the fact they words have a few common concepts (such as agile methodologies vs scrum vs kanban). However, if you have a good pretrained encoder, this similarities will be captured.

In the second and next iteration, you can now use the encoder to check if similarities between skills are flagged properly, using the for example cosine distance. You can then automatically label the similarities and eventually correct the labels. Once you have sufficient label, you can now retrain the encoders on your own (smaller) dataset.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How should my training data looks like at the end? $\endgroup$
    – user_12
    Commented Nov 16, 2020 at 8:16
  • $\begingroup$ @user_12 I the first iteration this is just a mapping from keywords to vectors. IF you have a lot of computational power, you could use the tensorflow encoders rights away to key-words will be directly transformed into vectors. I would recommend however to create a separate batch job and first create a look-up table with words to vector table. In second iteration, you will extend the table with two additional columns: label and label probability. The labels will be created by your data scientists after clustering similar vectors. $\endgroup$
    – Benoit Descamps
    Commented Nov 16, 2020 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.