As part of big data analysis project, I'm working on,
I need to perform PCA on some data, using cloud computing system.
In my case, I'm using Amazon EMR for the job and Spark in particular.
Leaving the "How-to-perform-PCA-in-spark" question aside, I want to get an understanding of how things work behind the scenes when it comes to calculating PCs on cloud-based architecture.
For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.
When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.
How then is the covariance matrix calculated in such case when each node have only partial data?
This is just an example. I'm trying to find some paper or documentation explaining all this behind-the-scenes voodoo, and couldn't find anything good enough for my needs (probably my poor google skills).
So I can basically summarize my question(s) \ needs to be the following:
1. How distributed PCA on cloud architecture works
Preferably some academic paper or other sorts of explanation which also contains some visuals
2. Spark implementation of D-PCA
How does Spark do it? Do they have any 'twist' in their architecture to do it more efficiently, or how does the RDD objects usage contribute to improving the efficiency? etc.
A presentation of even an online lesson regarding it would be great.
Thanks in advance to anyone who can provide some reading material.