I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' .corr()
function.
However, on such a large dataset, calculating the correlation matrix takes about 20 minutes. Is there any way to speed up the calculation?
PCA
model to reduce the number of columns. $\endgroup$PCA
simply reduces a large number of columns to a specified number of columns such that each column is a combination of older combinations and they can explain the variance in descending order. So, the firstPCA
column will explain the variance amongst the older columns most, and then comes the secondPCA
element, and so on. You can runPCA
in sklearn.decomposition.PCA $\endgroup$