6
$\begingroup$

I am trying to be sure that can I scrape government data from several websites if there is no mention of any commercial usage? I am willing to scrape US navy data(Link) and Canada Industrial Data (Link) and not sure if I should do. I personally don't hesitate to scrape govt. data but sometimes it may take the whole company in trouble specially in US. I need some valuable suggestions how to classify any website whether we should scrape or not.

$\endgroup$
1
  • $\begingroup$ Why not asking them? ("Contact us") $\endgroup$
    – tagoma
    Dec 12, 2017 at 21:58

1 Answer 1

4
$\begingroup$

Short answer: It all depends on how you plan to use the data you scraped. If you plan to upload it to your website, that is generally not OK since content can be copy write protected and you could get sued. If you plan to just use the data for personal use offline you should generally not have a problem.

I suggest you take a look at these links:

Those are the privacy pages found at the bottom of the links you provided.

You will find all the information you need there, including email addresses to which you can write to if you have any questions.

Though to help you out a bit, both of the sites you provided state that most of the content or all of it is copy write protected.

In conclusion, ask permission if you are going to re post the data somewhere on the internet. Websites that allow you to do so clearly state it in their privacy policies or terms of use.

$\endgroup$
2
  • $\begingroup$ makes sense, I agree. I was just wondering if we have some generalized method to know which all sites to scrape. Thanks anyways. $\endgroup$
    – Hari_pb
    Dec 14, 2017 at 4:51
  • $\begingroup$ You are allowed to scrape your own sites, but proceed to others with caution. For instance, a site owner might have paid a lot of money for unique content (text, pictures, etc.) and doesn't wan't you to download it or re use it. Site owners can set up the server to restrict fetching of certain items, but that doesn't mean that if there is no restriction that you are allowed to download it. This article is a good read on the topic, I suggest you read it. $\endgroup$
    – Ivan86
    Dec 14, 2017 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.