10
$\begingroup$

I am trying to learn web scraping using Python by myself as part of an effort to learn data analysis. I am trying to scrape imdb webpage whose url is the following: http://www.imdb.com/search/title?sort=num_votes,desc&start=1&title_type=feature&year=1950,2012

I am using BeautifulSoup module. Following is the code I am using:

r = requests.get(url) # where url is the above url    
bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
    title = movie.find('a').contents[0]
    genres = movie.find('span','genre').findAll('a')
    genres = [g.contents[0] for g in genres]
    runtime = movie.find('span','runtime').contents[0]
    year = movie.find('span','year_type').contents[0]
    print title, genres,runtime, rating, year

I am getting the following outputs:

The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. (1994)

Using this code, I could scrape title, genre, runtime,and year but I couldn't scrape the imdb movie id,nor the rating. After inspecting the elements (in chrome browser), I am not being able to find a pattern which will let me use similar code as above.

Can anybody help me write the piece of code that will let me scrape the movie id and ratings ?

$\endgroup$
  • 1
    $\begingroup$ I edited your code a bit but it fails because rating isn't defined. If you fix that, you could also add from BeautifulSoup import BeautifulSoup, and import requests. And why not also show url="http://etc" so we don't have to do that for ourselves? $\endgroup$ – Spacedman May 1 '15 at 15:40
  • 1
    $\begingroup$ Just in case: opendata.stackexchange.com/questions/1073/… $\endgroup$ – Anton Tarasenko May 5 '15 at 19:25
12
$\begingroup$

Instead of scraping, you might try to get the data directly here: http://www.imdb.com/interfaces. It looks like they have data available via ftp for movies, actors, etc.

$\endgroup$
  • 2
    $\begingroup$ @Gred Thatcher, Thanks for the link. This project is part of a learning endeavor on web scraping and hence all these troubles. -:) $\endgroup$ – user62198 May 1 '15 at 21:47
8
$\begingroup$

I have been able to figure out a solution. I thought of posting just in case it is of any help to anyone or if somebody wants to suggest something different.

bs = BeautifulSoup(r.text)
for movie in bs.findAll('td','title'):
    title = movie.find('a').contents[0]
    genres = movie.find('span','genre').findAll('a')
    genres = [g.contents[0] for g in genres]
    runtime = movie.find('span','runtime').contents[0]
    rating = movie.find('span','value').contents[0]
    year = movie.find('span','year_type').contents[0]
    imdbID = movie.find('span','rating-cancel').a['href'].split('/')[2]
    print title, genres,runtime, rating, year, imdbID

The output looks like this:

The Shawshank Redemption [u'Crime', u'Drama'] 142 mins. 9.3 (1994) tt0111161
$\endgroup$
2
$\begingroup$

You can get everything from div with class="rating rating-list"

All you need to do is retrive attribute id: [id="tt1345836|imdb|8.5|8.5|advsearch"] When you have this content, you split this string by '|', and you get: 1. parameter: movie id 3. parameter: movie score

$\endgroup$
  • $\begingroup$ Thanks. @Matic DB...i was able to get the id ..Below is my solution $\endgroup$ – user62198 Apr 16 '15 at 15:53
2
$\begingroup$

As a bit of general feedback, I think you would do well to improve your output format. The problem with the format as it stands is there is not a transparent way to programmatically get the data. Consider instead trying:

print "\t".join([title, genres,runtime, rating, year])

The nice thing about a tab delimited file is that if you end up scaling up, it can easily be read into something like impala (or at smaller scales, simple mySql tables). Additionally, you can then programatically read in the data in python using:

 line.split("\t")

The second bit of advice, is I would suggest getting more information than you think you need on your initial scrape. Disk space is cheaper than processing time, so rerunning the scraper every time you expand your analytic will not be fun.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.