4
$\begingroup$

Is there a recommended approach for storing processed data for testing new data products?

Basically, I'd like to have a system where a data scientist or an analyst could think of a new data product to present to users, do the data processing to create it, and then put it in a data store that our application can then access easily.

What I'm not sure about is what kind of data store would be good for this type of "testing" use case. Since it would need to be flexible enough to handle different types of data products, like aggregates, windowed data, etc. And ideally it wouldn't require a huge instrumentation process to try out new things.

Improve this question
$\endgroup$
3
  • $\begingroup$ This system wouldn't do any processing itself? Just a repository of data sets like CKAN? $\endgroup$
    – Spacedman
    Commented Apr 8, 2015 at 17:57
  • 1
    $\begingroup$ You could always spin up a NoSQL Environment that allows analyst and other data guys to drop data in and play with. Then when data products are discovered, it can be transitioned to the production system that either NoSQL or even a RDBMS. We considered Hadoop because it allows less ETL development and has additional components such as HIVE and the works to play with. $\endgroup$
    – Glen Swan
    Commented Apr 8, 2015 at 22:52
  • 2
    $\begingroup$ From a storage perspective, processed data is no different from unprocessed one, so I think that any storage approach is appropriate, as long as it satisfies other requirements (compatibility with other systems, software, business processes, APIs, etc.), all focused on your IT environment. $\endgroup$
    – Aleksandr Blekh
    Commented Apr 13, 2015 at 4:05

1 Answer 1

Reset to default
2
$\begingroup$

You might try Azure Table Storage. Since you can't lock yourself down to a specific schema (since one data product might be aggregates whereas another might be time series or something else), Azure Table storage would give you the flexibility of storing data from multiple sources, each having their own format.

This would also lend itself to making a system highly scalable, as you could use Azure Service Bus in conjunction with Azure Table Storage.

You might check out this tutorial at Pluralsight, Applied Windows Azure, as it shows a number of examples, one using Table Storage and Service Bus, another using Hadoop, and I suspect that some of these might match the extensibility you are looking for.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.