# How to scrape imdb webpage?

I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage.

I am using BeautifulSoup module. Following is the code I am using:

r = requests.get(url) # where url is the above url
bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
title = movie.find('a').contents[0]
genres = movie.find('span','genre').findAll('a')
genres = [g.contents[0] for g in genres]
runtime = movie.find('span','runtime').contents[0]
year = movie.find('span','year_type').contents[0]
print title, genres,runtime, rating, year


I am getting the following outputs:

The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)


Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.

Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?

Instead of scraping, you might try to get the data directly here. It looks like they have data available via ftp for movies, actors, etc.

• @Gred Thatcher, Thanks for the link. This project is part of a learning endeavor on web scraping and hence all these troubles. -:) – user62198 May 1 '15 at 21:47

I have been able to figure out a solution. I thought of posting just in case it is of any help to anyone or if somebody wants to suggest something different.

bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
title = movie.find('a').contents[0]
genres = movie.find('span','genre').findAll('a')
genres = [g.contents[0] for g in genres]
runtime = movie.find('span','runtime').contents[0]
rating = movie.find('span','value').contents[0]
year = movie.find('span','year_type').contents[0]
imdbID = movie.find('span','rating-cancel').a['href'].split('/')[2]
print title, genres,runtime, rating, year, imdbID


The output looks like this:

The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. 9.3 (1994) tt0111161


As a bit of general feedback, I think you would do well to improve your output format. The problem with the format as it stands is there is not a transparent way to programmatically get the data. Consider instead trying:

print "\t".join([title, genres,runtime, rating, year])


The nice thing about a tab delimited file is that if you end up scaling up, it can easily be read into something like impala (or at smaller scales, simple mySql tables). Additionally, you can then programatically read in the data in python using:

 line.split("\t")


The second bit of advice, is I would suggest getting more information than you think you need on your initial scrape. Disk space is cheaper than processing time, so rerunning the scraper every time you expand your analytic will not be fun.

You can get everything from div with class="rating rating-list"

All you need to do is retrive attribute id: [id="tt1345836|imdb|8.5|8.5|advsearch"] When you have this content, you split this string by '|', and you get:

1. parameter: movie id
2. parameter: movie score
• Thanks. @Matic DB...i was able to get the id ..Below is my solution – user62198 Apr 16 '15 at 15:53