0
$\begingroup$

In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot.

The value prop is that you move the low value (and less used) data from EDW to Hadoop and then save on expensive EDW storage by using Hadoop.

So in my company say we keep 10 years of data on EDW and say we don't use anything below 2 years very actively. So I move 8 years of data from EDW to Hadoop.

I can setup an Impala (or equivalent) product to query the 8 years of data as well.

But the problem is how do you order, sort a query which requires some data form EDW and some data from Impala?

because this kind of grouping, sorting ordering etc will have to be done in memory and the apps which queried the EDW were not written for operations and don't have the capacity as well to sort, group and process so much of data.

So how are people implementing the Active Archive?

Improve this question
$\endgroup$
1
$\begingroup$

I think the idea is that you have all data in HDFS, and query it with Impala, and also keep some small amount of data in your data warehouse. That is, keep all 10 years in Hadoop, and also 2 years in an EDW. The cost of also having the 2 years in Hadoop is small.

Improve this answer
$\endgroup$
  • $\begingroup$ so how do apps know where to query. should they choose impala with 10 years of data? or choose oracle with 2 years of data? $\endgroup$ – Knows Not Much Jan 24 '15 at 23:11
  • $\begingroup$ also, both systems would need business friendly data views. so won't this approach lead to duplicate development? $\endgroup$ – Knows Not Much Jan 24 '15 at 23:11
  • 1
    $\begingroup$ I presume that there are different use cases for the data warehouse versus active archive, so no I don't see that there is necessarily any duplication. You query the data source that fits your use case. $\endgroup$ – Sean Owen Jan 24 '15 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.