I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage whose url is the following: http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012
I am using BeautifulSoup module. Following is the code I am using:
r = requests.get(url) # where url is the above url bs = BeautifulSoup(r.text) for movie in bs.findAll('td','title'): title = movie.find('a').contents genres = movie.find('span','genre').findAll('a') genres = [g.contents for g in genres] runtime = movie.find('span','runtime').contents year = movie.find('span','year_type').contents print title, genres,runtime, rating, year
I am getting the following outputs:
The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)
Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.
Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?