I have a bunch of URLs in a text file like-


What's the best way to go about extracting top 10 keywords from just these URLs (no parsing the webpages)?

I'm aware of TF-IDF but that usually requires a title and a body, can I still use it here? Or are there any other approaches (e.g. TextRank) that would work better here?

  • $\begingroup$ Ideally what would you expect the keywords to be in your examples? $\endgroup$ – Erwan Nov 26 '19 at 1:33
  • $\begingroup$ @Erwan I would expect the keywords to be a mixture of domain names and words from page titles. For the URLs above, the keywords could be marketing, content, forbes, .... Maybe the domain names could be the title and the page URLs be the body if I want to do TF-IDF? Or maybe just TextRank would work better? I'm not sure.. $\endgroup$ – kev Nov 26 '19 at 16:09
  • $\begingroup$ Treat the URL's like a normal text. Start with tokenization, use stemming or lemmatization if you want, and then apply something like TF-IDF on the resulting words. $\endgroup$ – louic Nov 26 '19 at 17:04

urllib parse seems like the function for you. With this you are able to extract keywords from the net location and the path separately if you desire to process them separately or even if you want to join them back again later.

The result should look something like this:

from urllib.parse import urlparse

o = urlparse('https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts')

ParseResult(scheme='https', netloc='www.forbes.com:443', path='sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts',
           params='', query='', fragment='')

A second step will have to do with string parsing and string splits so something like


Will work just fine to split the path of the example's URLs into words. Also, remember to transform each word into lowercase. After that stemming is a good idea so that all related terms are grouped into the same category. It's not necessary but it's a good idea.

Finally counting the occurrences of the words will give you the rank of the top "keywords" in those URLs.

| improve this answer | |
  • $\begingroup$ say I got result = ['how', 'to', 'use', 'content', 'marketing', 'to', 'boost', 'your', 'recruiting', 'efforts', 'forbes'] for the first URL and also append all the other URLs' words to this list and remove the stopwords and do stemming. So you suggest I just do a word count on this final list to get the keywords? $\endgroup$ – kev Nov 27 '19 at 17:24
  • $\begingroup$ Yes, pretty much. I would try a count with stemming and another without. They probably will look similar but there is the chance that one is a more accurate representation than the other, depending on your dataset of course. $\endgroup$ – wacax Dec 4 '19 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.