In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot.
The value prop is that you move the low value (and less used) data from EDW to Hadoop and then save on expensive EDW storage by using Hadoop.
So in my company say we keep 10 years of data on EDW and say we don't use anything below 2 years very actively. So I move 8 years of data from EDW to Hadoop.
I can setup an Impala (or equivalent) product to query the 8 years of data as well.
But the problem is how do you order, sort a query which requires some data form EDW and some data from Impala?
because this kind of grouping, sorting ordering etc will have to be done in memory and the apps which queried the EDW were not written for operations and don't have the capacity as well to sort, group and process so much of data.
So how are people implementing the Active Archive?