0
$\begingroup$

I am dealing with a data matrix in which most of the variables are binary or multilevel response. I would like to perform the MDS algorithm and for that, I need to calculate the distance matrix first. My dataset contains about 4,000,000 individuals, so computationally it exceeds the capabilities of my computer. How could I compute this matrix efficiently?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

Some high memory processes could be solved in hard disk instead of RAM, but if it takes too long, you should improve your algorithm as much as possible indeed.

To improve the algorithm, could you give a small sample?

To improve the hardware use, you could make a small experiment with 100 000 individuals, then 200 000 and see what kind of increase you have, checking the RAM and the time used.

If it is linear, you can interpolate to 4 million. But if it is exponential, it might be impossible to solve.

That’s why you should check if the problem could be solved or not, and what would be the minimum hardware.

On the other hand, low level operations and data structures (like sets in Python) could lower the memory use greatly.

Web services like Paperspace, gcloud, etc. could be also helpful because you can rent powerful hardware for interesting prices. Some GPUs like Google’s C2 can solve many problems with very high memory and calculation requirements. https://cloud.google.com/compute/docs/machine-types

$\endgroup$
1
  • $\begingroup$ Does it answer your question? If not, please let me know. $\endgroup$ Sep 9, 2022 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.