This is a bit of a stretch topic, but I find myself often browsing gear websites (like, Gear Patrol and Uncrate) for birthday and gift ideas. I find it cumbersome to click through each and every article looking for items that are either within my price range or within my price range && category of item that I'm looking for.

I know that I can go to Amazon and set specific filters for items that I am looking for, but I enjoy receiving suggestions from websites that are known to give quality product recommendations.

From a data mining / web scraping standpoint, does anyone know of any resources of tutorials online that provide guidance for similar products?

Is R or Python better suited for web scraping applications?

Any insight into my idea would be greatly appreciated :)


1 Answer 1


I would look into the Python package BeautifulSoup. It parses HTML documents into a tree structure and allows for all kind of filtering and manipulation of the tree. I've taken a look at the Gear Patrol website and I would advice these steps:

Scrape the front page looking for articles. By taking a look using a web inspector like the one in Chrome (Ctrl+Shift+I) you can look at the HTML structure. Looking there we can see that the links to the articles are in:

body > div:content > div:home-this-week > div:posts hfeed > article > div:body-wrap

Now you look for all article tags and create a list of the corresponding URLs. You can use the link of the pagination to retrieve more article links or use their URL structure to make an educated guess where more articles are.

Now that we have a list of URLs, iterate over them and parse these pages in BeautifulSoup to retrieve the necessary information in a similar fashion. I advice taking breaks of a few seconds in between requests for new pages to not get banned from their server and not overly disrupt it. There are a lot of tutorials available how to do webscraping using BeautifulSoup and this site looks like an easy victim.

  • $\begingroup$ Thanks for the fantastic response, I will look into BeautifulSoup! $\endgroup$
    – Gary
    Commented May 18, 2016 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.