1
$\begingroup$

I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing on. The code I'm using returns the title and a description of the web page. I've also extracted the links for the results. Here is the code I used:

from bs4 import BeautifulSoup
import urllib, urllib2

def bing_search(query):
    address = "http://www.bing.com/search?q=%s" % (urllib.quote_plus(query))

    getRequest = urllib2.Request(address, None, {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 Chrome/65.0.3325.162 Safari/537.36'})

    urlfile = urllib2.urlopen(getRequest)
    htmlResult = urlfile.read(200000)
    urlfile.close()

    soup = BeautifulSoup(htmlResult)

    [s.extract() for s in soup('span')]
    #unwantedTags = ['a', 'strong', 'cite']
    #for tag in unwatedTags:
        #for match in soup.findAll(tag):
           # match.replaceWithChildren()

    results = soup.findAll('li', {"class" : "b_algo" })
    for result in results: 
        print "# TITLE: " + str(result.find('h2')).replace(" ", " ") + "\n#"
        print "# DESCRIPTION: " + str(result.find('p')).replace(" ", " ")
        print "# ___________________________________________________________\n#"

    return results

if __name__ == '__main__':
    links = bing_search('rasmalai')

Now that I have the links, web page title, and a short description, I want to extract keywords using NLP. In the end, I'd like to produce a CSV file with the dish name and associated keywords. Could someone guide me to some resources on how to do this part?

Thank you so much in advance.

$\endgroup$
3
  • 1
    $\begingroup$ Can you specify what do you mean by extracting keywords using NLP? Do you want to produce keywords based on description? $\endgroup$ Jun 23, 2018 at 11:57
  • $\begingroup$ @DanielChepenko yes, that's exactly what I mean! $\endgroup$ Jun 23, 2018 at 15:49
  • $\begingroup$ blog.kbresearch.nl/2016/01/11/… Take a look at this post $\endgroup$ Jun 23, 2018 at 16:30

1 Answer 1

2
$\begingroup$

NLTK

A great starting point for keyword extraction is the NLTK (natural language toolkit) library. To extract keywords, you probably need to tokenize your data, breaking each word out into a token, and ignore the most common or unimportant words known as "stopwords". Assuming you're searching for keywords across a large number of query results, identify the most important terms in each document using TF-IDF (term frequency–inverse document frequency). There are tools and tutorials for this in the NLTK documentation. Sort the resulting token-scores, choose the highest scoring tokens, and these are a good start at your keywords.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.