I am picturing something like the following:

  1. List internet sources you want to acquire data from.
  2. Study the source code structure for each site. Try to build an “extractor” for each individual site that is as perfectly tailored to that site as possible. A very developed testing framework should help you immediately detect when one of the extractors has broken due to any site changes or site features.
  3. Run the extractors as often as is reasonable, ie for the needs of the application, according to the site terms of service, etc.
  4. Choose a data model in advance so that the extractors save the data in a common format.
  5. Run the extractor from a cloud server and save the data to a cloud database.
  6. Design an API to query the structured data from the database.

Does that sound like a standard or best-practices approach? Is there a common directory structure for this type of project, so I can just set it all up according to a common workflow?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.