1
$\begingroup$

I have search keywords in one of my database table. These are the keywords searched by users on a website. My requirement is to find the top 100 search keywords after consolidating various similar search keywords.

e.g. COVID19, Covid 19, Corona, Corona virus, covid-19 are all related to corona and it should be grouped under Corona.

OR

blockchain, BLOCKCHAIN, Blockchain business, B L O C K C H A I N are all related to Blockchain so it should be grouped under Blockchain category.

I tried to do this using fuzzywuzzy python library but fuzzywuzzy needs a list to be compared with the search keywords. And unfortunately I don't have that list because it can be anything based on user's search.

Any suggestions on the approach would really help.

$\endgroup$
6
  • $\begingroup$ Did you try from difflib import get_close_matches? I also assume you are removing space/punctuation and making it all in same cases before processing?There is stemming/lemmatization also in nltk. I am not sure it would be helpful in this case though. $\endgroup$ May 21 '20 at 9:53
  • $\begingroup$ I haven't tried but is it not similar to fuzzywuzzy where we can compare two words and see if they are close match ? And how will it help in the cases like Artificial Intelligence is same as AI. And Covid is same as Corona. Lets say I have 100 words, then do you suggest comparing first word with rest 99 and so on. And yes I am doing preprocessing of the search keywords. $\endgroup$
    – Akash
    May 21 '20 at 10:00
  • $\begingroup$ It is similar. No I am not suggesting that you compare each word that would not be efficient. What comes to mind is finding similarities.May be clustering ?K-Means?I never tried it in text but sounds suitable.if this helps at all. $\endgroup$ May 21 '20 at 10:12
  • $\begingroup$ may be this would help-stats.stackexchange.com/questions/123060/… $\endgroup$ May 21 '20 at 10:26
  • $\begingroup$ while I can go through the link, but looks like the post is very old and may be we have a better approach to do it now ? Thank you for finding that link for me. $\endgroup$
    – Akash
    May 21 '20 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.