-3
$\begingroup$

I want to scrape some data from a website. I have used import.io but still not much satisfied.. can any of you suggest about it.. whats the best tool to get the unstructured data from web

$\endgroup$

closed as unclear what you're asking by damienfrancois, Sean Owen May 26 '15 at 6:36

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    $\begingroup$ Why not point us to the page that you are trying to scrape? The "best tool" is hard to define otherwise. $\endgroup$ – Spacedman Aug 20 '14 at 15:55
  • 2
    $\begingroup$ On StackExchange sites, when you say "doesn't work" or "not satisfied", you really have to say what that means. Was it too slow? expensive? feature missing? Otherwise, how do you expect a suggestion for something "better"? $\endgroup$ – Sean Owen Aug 21 '14 at 15:49
2
$\begingroup$

Try BeautifulSoup - http://www.crummy.com/software/BeautifulSoup/

From the website "Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping." I have no personally used it, but it often comes up in regards to a nice library for scraping. Here's a blog post on using it to scrape Craigslist http://www.gregreda.com/2014/07/27/scraping-craigslist-for-tickets/

$\endgroup$
1
$\begingroup$

You don't mention what language you're programming in (please consider adding it as a tag), so general help would be to seek out a HTML parser and use that to pull the data. Some web sites can have simply awful HTML code and can be very difficult to scrape, and just when you think you have it...

A HTML parser will parse all the html and allow you to access it in a structured sort of way, whether that's from an array, an object etc.

$\endgroup$
  • $\begingroup$ Some sites create their HTML from Javascript and then you need a Javascript-powered scraper (Selenium). Or you reverse engineer their JSON API and get the data directly... $\endgroup$ – Spacedman Aug 21 '14 at 15:20
  • $\begingroup$ some sites do, but the majority do not. As OP didn't specify, one assumes it's a 'normal' site. $\endgroup$ – user3791372 Aug 21 '14 at 15:22
0
$\begingroup$

Ruby together with Nokogiri allows to access HTML and XML documents via XPath and CSS selectors. Here is a tutorial.

$\endgroup$
0
$\begingroup$

You don't need a tool and I don't recommend you use one.

Convert the html to well-formed XML (XHTML) - I recommend the tagsoup.

Once you've done that the data is just another XML feed and you can write an XSLT transformation (or XQuery) to access and pull out the data you want in the format you want.

That might mean learning XSLT/XQuery if you don't already know it but you will be learning skills that (unlike scraping tools) have multiple rather than just than one useful application.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.