I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to achieve this?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ When you say 'not able to to it' do you mean you don't know how to technically implement it or do you run into errors/other issues? $\endgroup$– OxbowerceSep 22, 2021 at 7:37
-
$\begingroup$ I get MemoryError: Unable to allocate 340. GiB for an array with shape (307180, 148485) and data type float64 $\endgroup$– user16584277Sep 22, 2021 at 7:42
-
$\begingroup$ I would have expected as much, I am not sure there are good solutions other than changing the data type to a lower precision (e.g. float32 or even lower if possible) or maybe using a sparse matrix if you're working with sparse data. $\endgroup$– OxbowerceSep 22, 2021 at 7:51
-
2$\begingroup$ Maybe dask. $\endgroup$– noeSep 22, 2021 at 8:32
-
1$\begingroup$ What transformation? Some operations don't require the entire matrix to be in memory, but rather you can load in columns/rows as needed and persist the rest to disk. $\endgroup$– AndySep 22, 2021 at 16:09
|
Show 1 more comment
4 Answers
$\begingroup$
$\endgroup$
1
You can also buy some GPU's which will always help you to make up for the low memory allocation.
Cloud services will help as well but the variable costs are too high if your objective is to work on high dimension matrices
-
1$\begingroup$ How does buying GPUs help in allocating a matrix of 340 GiB? $\endgroup$ Sep 22, 2021 at 20:28
$\begingroup$
$\endgroup$
2
I don't know if this is your job work or your personal project but cloud services can help you. Create a free account on Azure (you get 200$ worth free credit for 1 month) and the account is free for1 year.
You can run your project there using Azure Machine Learning/AutoML/Python SDK whatever you choose. Use the free 200$ credit in 1 month for any kind of large scale project which requires large computational power or large memory requirement. The link will give you an idea about the computational power and memory capacities on Azure.
-
1$\begingroup$ So on Azure you can allocate a matrix of 340 GiB? Unlikely! $\endgroup$ Sep 22, 2021 at 20:28
-
$\begingroup$ docs.microsoft.com/en-us/azure/virtual-machines/sizes-memory Yes you can. The Mv2 series offers upto 11.4Tb. $\endgroup$– spectreSep 23, 2021 at 4:55
$\begingroup$
$\endgroup$
Are many of the entries in the matrix zero? In this case, you can often deal with large matrices without using large amounts of memory. Sparse matrix data structures exist and so do algorithms for doing arithmetic with them.
scipy includes support for sparse matrices. https://docs.scipy.org/doc/scipy/reference/sparse.html
$\begingroup$
$\endgroup$
You can use Dask array, in which you can create essentially any size array and apply some transformation. What it basically does is load a chunk of the array that can be fit into memory process it and then save it. Check the following link :
[https://examples.dask.org/array.html][1]
