2
$\begingroup$

I have a bunch of URLs in a text file like-

https://www.mycustomer.com/marketing/technology/how-marketers-are-over-simplifying-b2b-buyer-behaviour
https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts
https://www.forbes.com/sites/forbesagencycouncil/2017/02/03/video-marketing-the-future-of-content-marketing
https://www.searchenginejournal.com/seo-content-marketing-strategy/258253
...

What's the best way to go about extracting top 10 keywords from just these URLs (no parsing the webpages)?

I'm aware of TF-IDF but that usually requires a title and a body, can I still use it here? Or are there any other approaches (e.g. TextRank) that would work better here?

Improve this question
$\endgroup$
3
  • $\begingroup$ Ideally what would you expect the keywords to be in your examples? $\endgroup$
    – Erwan
    Nov 26, 2019 at 1:33
  • $\begingroup$ @Erwan I would expect the keywords to be a mixture of domain names and words from page titles. For the URLs above, the keywords could be marketing, content, forbes, .... Maybe the domain names could be the title and the page URLs be the body if I want to do TF-IDF? Or maybe just TextRank would work better? I'm not sure.. $\endgroup$
    – kev
    Nov 26, 2019 at 16:09
  • $\begingroup$ Treat the URL's like a normal text. Start with tokenization, use stemming or lemmatization if you want, and then apply something like TF-IDF on the resulting words. $\endgroup$
    – Louic
    Nov 26, 2019 at 17:04

2 Answers 2

Reset to default
1
$\begingroup$

urllib parse seems like the function for you. With this you are able to extract keywords from the net location and the path separately if you desire to process them separately or even if you want to join them back again later.

The result should look something like this:

from urllib.parse import urlparse

o = urlparse('https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts')

ParseResult(scheme='https', netloc='www.forbes.com:443', path='sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts',
           params='', query='', fragment='')

A second step will have to do with string parsing and string splits so something like 

.split("/")
.split("-")

Will work just fine to split the path of the example's URLs into words. Also, remember to transform each word into lowercase. After that stemming is a good idea so that all related terms are grouped into the same category. It's not necessary but it's a good idea.

Finally counting the occurrences of the words will give you the rank of the top "keywords" in those URLs.

Improve this answer
$\endgroup$
2
  • $\begingroup$ say I got result = ['how', 'to', 'use', 'content', 'marketing', 'to', 'boost', 'your', 'recruiting', 'efforts', 'forbes'] for the first URL and also append all the other URLs' words to this list and remove the stopwords and do stemming. So you suggest I just do a word count on this final list to get the keywords? $\endgroup$
    – kev
    Nov 27, 2019 at 17:24
  • $\begingroup$ Yes, pretty much. I would try a count with stemming and another without. They probably will look similar but there is the chance that one is a more accurate representation than the other, depending on your dataset of course. $\endgroup$
    – wacax
    Dec 4, 2019 at 20:38
0
$\begingroup$

Using Query Function formulas in Google Sheets helps in extracting the keywords from any URL so that is 1 such method that can be used efficiently.

For Example:

https://play.google.com/store/search?q=messenger%20app&c=apps

https://play.google.com/store/search?q=torch&c=apps

https://play.google.com/store/search?q=personal%20loan%20app&c=apps

https://play.google.com/store/search?q=pnr%20status%20in%20india&c=apps

These are the URLs I need keywords from then Here is one method you could try....

If the first cell that has the link is in A2, put this into cell B2, and drag down:

=REGEXREPLACE(INDEX(SPLIT(INDEX(SPLIT(A2,"="),2),"&"),1),"%20"," ")

alternatively, for a single cell arrayformula (so no need to drag down), just put this into B2:

=ARRAYFORMULA(IFERROR( INDEX(SPLIT(REGEXEXTRACT(REGEXREPLACE(SUBSTITUTE(A5:A, "&", "="),"%20"," ")&"(", "\=(.+)\="), "="),,1)))

Alternatively the formula below works fine too.

=SPLIT(INDEX(SPLIT(INDEX(SPLIT(INDEX(SPLIT(B2, "/"),4),"="),2),"&"),1),"%20")
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.