2
$\begingroup$

I want to know if there is any way that I can crawl customer reviews for particular products from amazon without being blocked. At the moment, my crawler is blocked after a few times. Any idea will be appreciated.

$\endgroup$
4
  • 2
    $\begingroup$ Pace your requests, adjusting the frequency to determine what their limits are. Use an archived dataset instead if it serves your purposes. $\endgroup$
    – Emre
    May 25 '17 at 19:20
  • $\begingroup$ Thank you for your suggestion. I have paused my crawler for 10s sec each time and yet being blocked before the job is done. So, I'm looking for what can be added to my crawler to get the job done. $\endgroup$
    – bensw
    May 26 '17 at 1:34
  • 1
    $\begingroup$ You are not waiting enough then. Keep backing off until you find an acceptible rate. You might also need to add noise delay to make your request distribution more human. You can use multiple servers to speed things up. $\endgroup$
    – Emre
    May 26 '17 at 1:46
  • $\begingroup$ Add some randomization to all your delay. And try to do more variable search queries. Like dont query in a very repeatable way. Also don't open pages sequential. That is too obvious. $\endgroup$
    – JahKnows
    May 26 '17 at 13:00
3
$\begingroup$

You are getting blocked because people do not want to waste server bandwidth on someone who is trying to exploit it without bringing significant profits.

Try to make your crawling less predictable.

Slow down the frequency with which you ping the server and vary the actions of your crawler. This will make it harder to detect as it will act less predictably and may be wrongfully identified as being a very quick human.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your suggestion. I'll try that. If it works, I'll get back to you. $\endgroup$
    – bensw
    May 26 '17 at 1:35
0
$\begingroup$

Amazon will detect the scraper from its fast and regular actions, and the same IP. Normally, scraping automation tools could skip its block by rotating the IPs and slow down the actions. Our product Octoparse Cloud Extraction could solve this problem with our hundreds of IPs and can break down the actions of the crawler to different servers.

$\endgroup$
1
  • 2
    $\begingroup$ Please note that you must disclose your affiliation in your answer: see the help center. $\endgroup$
    – Glorfindel
    Aug 30 '18 at 7:48
0
$\begingroup$

Based on my experience, you will need at least 1 proxy every 10 amazon requests, that means that if you want to crawl 1000 products, you are going to need 100 proxies to be on the safe side.

In the past I tried many services like luminati.io and proxyrack.com the problem both have is that they proxies end up being blocked and you have to get new ones which ends up being super expensive.

So I tried with proxycrawl.com which offered a price based on consumption and not on proxies and that worked much better as I don't have to care now about the amount of proxies, I just load the amazon comments.

So to be clear, if you want to use your own proxies, calculate 1 proxy every 10 products, otherwise just search for a company which can handle all that for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.