I want to scrape some data from a website. I have used import.io but still not much satisfied.. can any of you suggest about it.. whats the best tool to get the unstructured data from web
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ Why not point us to the page that you are trying to scrape? The "best tool" is hard to define otherwise. $\endgroup$– SpacedmanCommented Aug 20, 2014 at 15:55
-
2$\begingroup$ On StackExchange sites, when you say "doesn't work" or "not satisfied", you really have to say what that means. Was it too slow? expensive? feature missing? Otherwise, how do you expect a suggestion for something "better"? $\endgroup$– Sean OwenCommented Aug 21, 2014 at 15:49
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
Try BeautifulSoup - http://www.crummy.com/software/BeautifulSoup/
From the website "Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping." I have no personally used it, but it often comes up in regards to a nice library for scraping. Here's a blog post on using it to scrape Craigslist http://www.gregreda.com/2014/07/27/scraping-craigslist-for-tickets/
$\begingroup$
$\endgroup$
2
You don't mention what language you're programming in (please consider adding it as a tag), so general help would be to seek out a HTML parser and use that to pull the data. Some web sites can have simply awful HTML code and can be very difficult to scrape, and just when you think you have it...
A HTML parser will parse all the html and allow you to access it in a structured sort of way, whether that's from an array, an object etc.
-
$\begingroup$ Some sites create their HTML from Javascript and then you need a Javascript-powered scraper (Selenium). Or you reverse engineer their JSON API and get the data directly... $\endgroup$ Commented Aug 21, 2014 at 15:20
-
$\begingroup$ some sites do, but the majority do not. As OP didn't specify, one assumes it's a 'normal' site. $\endgroup$ Commented Aug 21, 2014 at 15:22
$\begingroup$
$\endgroup$
You don't need a tool and I don't recommend you use one.
Convert the html to well-formed XML (XHTML) - I recommend the tagsoup.
Once you've done that the data is just another XML feed and you can write an XSLT transformation (or XQuery) to access and pull out the data you want in the format you want.
That might mean learning XSLT/XQuery if you don't already know it but you will be learning skills that (unlike scraping tools) have multiple rather than just than one useful application.
