2
$\begingroup$

I'm using a dataset with roughly 460,000 rows and 1,300 columns. I'd like to reduce the number of columns by seeing which have the largest effect on score using pandas' .corr() function.

However, on such a large dataset, calculating the correlation matrix takes about 20 minutes. Is there any way to speed up the calculation?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Not directly related to your question, but you may run a PCA model to reduce the number of columns. $\endgroup$
    – Nuri Taş
    Nov 21, 2022 at 22:14
  • $\begingroup$ How does that work in Pandas? How does PCA let you know which columns to remove? Do I have to tell PCA my target column? $\endgroup$
    – Connor
    Nov 24, 2022 at 13:36
  • 1
    $\begingroup$ PCA simply reduces a large number of columns to a specified number of columns such that each column is a combination of older combinations and they can explain the variance in descending order. So, the first PCA column will explain the variance amongst the older columns most, and then comes the second PCA element, and so on. You can run PCA in sklearn.decomposition.PCA $\endgroup$
    – Nuri Taş
    Nov 24, 2022 at 13:51

1 Answer 1

Reset to default
2
$\begingroup$

You can use libraries with similar or identical pandas syntax, such as: dask, pandaralells, ray, modin. Each of these libraries allows all processor cores to work. Pandas often uses only 1 core. Dask and ray also allow you to work with big data.

It is also possible to select only part of the dataset. 460,000 is quite a lot, I think if you accidentally take half of this value, the result will be very similar if you take the entire dataset. Unfortunately, I cannot mathematically estimate how much difference there will be.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, a further question, I'm working on a closed system and it's not clear I'll be able to get those packages in a reasonable length of time. What are my other options if those packages aren't easily available to me? $\endgroup$
    – Connor
    Nov 23, 2022 at 13:21
  • 1
    $\begingroup$ If the calculations can be reused, then they can be saved to disk and not recalculated every time jupiter notebook runs. You can try to change the types to simpler ones, but I'm not sure that this will give a big advantage. For example, df['a']=df['a'].astype(numpy.float32) //a - float64. Also, for example, when loading from a database, types may be incorrect, for example varchar, which will give an object type in pandas. They also need to be converted. Nothing else comes to mind. $\endgroup$
    – Andrew
    Nov 24, 2022 at 9:23
  • $\begingroup$ Thank you! At the moment I'm limiting the columns I take in, but I guess in the future I'll have to find a more robust approach. $\endgroup$
    – Connor
    Nov 24, 2022 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.