1
$\begingroup$

What are the required stages for transferring data in datawarehouse into big data structure. Are there any tools and methods that support it? How to use the schema for such transformation, how to deal with different data types like facts and dimensions for instance. What is the criteria for data separation into machines, indexes or unique keys?

$\endgroup$
1
$\begingroup$

Based on your comments transitioning big data sets from on-premises systems to a cloud-based system is cumbersome and fraught with challenges. However, you can use Amazon RedShift:

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more.

  1. The first step is to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster.
  2. After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.

OR

  1. Use a Hadoop environment as the landing zone to pull in data from various sources, process it, and transfer the processed data to the existing data warehouse or other repositories.
  2. Explore scenarios of different ways to implement a landing zone. Learn about the architecture of the zone and the tools and techniques for integrating it with various environments.

enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks @Arjun. My question is from different perspective. You already have datawarehouse but because of explosion of data you can no longer use it and you want to transform the data into cloud, for instance. So the question is how to do it? $\endgroup$ – Avi Sep 13 '16 at 18:53
  • $\begingroup$ Hi avi, I knew this is the problem otherwise who takes a backward step from data warehouse to data structure. And above solution is for that so hadoop provide tools to transform data from warehouse to other repository by acting as a Landing Zone for data. $\endgroup$ – Arjun Chaudhary Sep 13 '16 at 18:57
  • $\begingroup$ Thanks again. Can you please elaborate more on the required steps. For instance, how to use the schema for such transformation, how to deal with different data types like facts and dimensions for instance. What is the criteria for data separation into machines, indexes or unique keys? $\endgroup$ – Avi Sep 13 '16 at 19:01
  • $\begingroup$ Yup I can but its a cumbersome steps. In IBM we use Amazon Redshift services to do this for big projects. I have included some common steps in above answer if you want to go for that but stiill for more info u can visit their website.. $\endgroup$ – Arjun Chaudhary Sep 13 '16 at 19:06
  • 1
    $\begingroup$ I understood what you are saying I may know a way but i have to read about it. Its interesting, I will get back to you with proper methodology. $\endgroup$ – Arjun Chaudhary Sep 13 '16 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.