Grouping profiles strings having the same words, but occurring out of order

Question

I have a dataframe containing a column of profile types, which looks like this:

0                                    Android Java
1                  Software Development Developer
2                            Full-stack Developer
3                      JavaScript Frontend Design
4                          Android iOS JavaScript
5                             Ruby JavaScript PHP

I've used NLP to fuzzy match similar profiles, which returned the following similarity dataframe:

left_side                       right_side                  similarity
7   JavaScript Frontend Design  Design JavaScript Frontend  0.849943
8   JavaScript Frontend Design  Frontend Design JavaScript  0.814599
9   JavaScript Frontend Design  JavaScript Frontend         0.808010
10  JavaScript Frontend Design  Frontend JavaScript Design  0.802881
12  Android iOS JavaScript      Android iOS Java            0.925126
15  Machine Learning Engineer   Machine Learning Developer  0.839165
21  Android Developer Developer Android Developer           0.872646
25  Design Marketing Testing    Design Marketing            0.817195
28  Quality Assurance           Quality Assurance Developer 0.948010

While this has helped, taking me from 478 unique profile to 461, what I'd want to focus on are profiles like this:

Frontend Design JavaScript  Design Frontend JavaScript

The only tool I've seen which looks to address this problem is difflib? My question is, what other techniques would be available so as to go through and standardize these profiles that consist of the same words, but out of order, to one standard string. So desired output would be, taking a string containing "Design", "Frontend" and "JavaScript" and replacing it with "Design Frontend JavaScript".

Right now, I'm merging my original dataframe with the similarity dataframe to replace all occurrences of profile string on the right_side with the left_side, but that means I'm replacing the right_side below ("Java Python Data Science") with the left_side below ("JavaScript Python Data Science").

53  JavaScript Python Data Science  Java Python Data Science

Any help would be greatly appreciated!!!

Answer 1

"fuzzy match" is a vague term for many different kind of string similarity measures. There are two main categories:

• Sequence-based methods, such as Levenshtein edit distance (and variants), Jaro-Winkler
• Bag-of-something methods, such as Cosine-TFIDF, Overlap, Jaccard, ...

Traditionally the first category is used at the level of characters (e.g. to match "abcdefg" with "abdfeg") and the second at the level of words (e.g. to match "the management of the team" with "the team management"), however there is no technical restrictions to keep it this way. The main difference between the two categories is whether or not they take the order of the units (e.g. words or characters) into account. In fact there are also some hybrid measures such as SoftTFIDF which combine both approaches.

Conclusion:

• For the problem of word order, a simple bag-of-words measure such as cosine or Jaccard would perfectly match two profiles.
• If more flexibility is needed, Soft-TFIDF or other variants would work.

The last part of the question is about selecting a standard string. In general there's no way to know what should be the standard string from the different variants themselves. A simple approach would be to pick the most common variant.