I want to know if there is any way that I can crawl customer reviews for particular products from amazon without being blocked. At the moment, my crawler is blocked after a few times. Any idea will be appreciated.
4
-
2$\begingroup$ Pace your requests, adjusting the frequency to determine what their limits are. Use an archived dataset instead if it serves your purposes. $\endgroup$– EmreMay 25, 2017 at 19:20
-
$\begingroup$ Thank you for your suggestion. I have paused my crawler for 10s sec each time and yet being blocked before the job is done. So, I'm looking for what can be added to my crawler to get the job done. $\endgroup$– benswMay 26, 2017 at 1:34
-
1$\begingroup$ You are not waiting enough then. Keep backing off until you find an acceptible rate. You might also need to add noise delay to make your request distribution more human. You can use multiple servers to speed things up. $\endgroup$– EmreMay 26, 2017 at 1:46
-
$\begingroup$ Add some randomization to all your delay. And try to do more variable search queries. Like dont query in a very repeatable way. Also don't open pages sequential. That is too obvious. $\endgroup$– JahKnowsMay 26, 2017 at 13:00
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
You are getting blocked because people do not want to waste server bandwidth on someone who is trying to exploit it without bringing significant profits.
Try to make your crawling less predictable.
Slow down the frequency with which you ping the server and vary the actions of your crawler. This will make it harder to detect as it will act less predictably and may be wrongfully identified as being a very quick human.
-
1$\begingroup$ Thank you for your suggestion. I'll try that. If it works, I'll get back to you. $\endgroup$– benswMay 26, 2017 at 1:35
$\begingroup$
$\endgroup$
1
Amazon will detect the scraper from its fast and regular actions, and the same IP. Normally, scraping automation tools could skip its block by rotating the IPs and slow down the actions. Our product Octoparse Cloud Extraction could solve this problem with our hundreds of IPs and can break down the actions of the crawler to different servers.
-
2$\begingroup$ Please note that you must disclose your affiliation in your answer: see the help center. $\endgroup$ Aug 30, 2018 at 7:48
$\begingroup$
$\endgroup$
Based on my experience, you will need at least 1 proxy every 10 amazon requests, that means that if you want to crawl 1000 products, you are going to need 100 proxies to be on the safe side.
In the past I tried many services like luminati.io and proxyrack.com the problem both have is that they proxies end up being blocked and you have to get new ones which ends up being super expensive.
So I tried with proxycrawl.com which offered a price based on consumption and not on proxies and that worked much better as I don't have to care now about the amount of proxies, I just load the amazon comments.
So to be clear, if you want to use your own proxies, calculate 1 proxy every 10 products, otherwise just search for a company which can handle all that for you.
