Is there a recommended approach for storing processed data for testing new data products?

Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product to present to users, do the data processing to create it, and then put it in a data store that our application can then access easily.

What I'm not sure about is what kind of data store would be good for this type of "testing" use case. Since it would need to be flexible enough to handle different types of data products, like aggregates, windowed data, etc. And ideally it wouldn't require a huge instrumentation process to try out new things.

  • $\begingroup$ This system wouldn't do any processing itself? Just a repository of data sets like CKAN? $\endgroup$ – Spacedman Apr 8 '15 at 17:57
  • 1
    $\begingroup$ You could always spin up a NoSQL Environment that allows analyst and other data guys to drop data in and play with. Then when data products are discovered, it can be transitioned to the production system that either NoSQL or even a RDBMS. We considered Hadoop because it allows less ETL development and has additional components such as HIVE and the works to play with. $\endgroup$ – Glen Swan Apr 8 '15 at 22:52
  • 2
    $\begingroup$ From a storage perspective, processed data is no different from unprocessed one, so I think that any storage approach is appropriate, as long as it satisfies other requirements (compatibility with other systems, software, business processes, APIs, etc.), all focused on your IT environment. $\endgroup$ – Aleksandr Blekh Apr 13 '15 at 4:05

You might try Azure Table Storage. Since you can't lock yourself down to a specific schema (since one data product might be aggregates whereas another might be time series or something else), Azure Table storage would give you the flexibility of storing data from multiple sources, each having their own format.

This would also lend itself to making a system highly scalable, as you could use Azure Service Bus in conjunction with Azure Table Storage.

You might check out this tutorial at Pluralsight, Applied Windows Azure, as it shows a number of examples, one using Table Storage and Service Bus, another using Hadoop, and I suspect that some of these might match the extensibility you are looking for.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.