Based on this answer, we know that we can perform build covariance matrix incrementally when there are too many observations, whereas we can perform randomised SVD when there are too many variables.
The answer provide are clear and helpful. However, what if we have a large amount of observations AND variables? e.g. 500,000 samples with 600,000 observations. In this case, the covariance matrix will be huge (e.g. 2,000 GB, assuming 8byte float, and if my calculation is correct) and will be impossible for us to fit it into memory.
In such scenario, is there anything that we can do to calculate the PCA, assuming we only want the top PCs (e.g. 15 PCs)?