0
$\begingroup$

I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to achieve this?

$\endgroup$
6
  • $\begingroup$ When you say 'not able to to it' do you mean you don't know how to technically implement it or do you run into errors/other issues? $\endgroup$
    – Oxbowerce
    Commented Sep 22, 2021 at 7:37
  • $\begingroup$ I get MemoryError: Unable to allocate 340. GiB for an array with shape (307180, 148485) and data type float64 $\endgroup$ Commented Sep 22, 2021 at 7:42
  • $\begingroup$ I would have expected as much, I am not sure there are good solutions other than changing the data type to a lower precision (e.g. float32 or even lower if possible) or maybe using a sparse matrix if you're working with sparse data. $\endgroup$
    – Oxbowerce
    Commented Sep 22, 2021 at 7:51
  • 2
    $\begingroup$ Maybe dask. $\endgroup$
    – noe
    Commented Sep 22, 2021 at 8:32
  • 1
    $\begingroup$ What transformation? Some operations don't require the entire matrix to be in memory, but rather you can load in columns/rows as needed and persist the rest to disk. $\endgroup$
    – Andy
    Commented Sep 22, 2021 at 16:09

4 Answers 4

3
$\begingroup$

You can also buy some GPU's which will always help you to make up for the low memory allocation.

Cloud services will help as well but the variable costs are too high if your objective is to work on high dimension matrices

$\endgroup$
1
  • 1
    $\begingroup$ How does buying GPUs help in allocating a matrix of 340 GiB? $\endgroup$ Commented Sep 22, 2021 at 20:28
3
$\begingroup$

I don't know if this is your job work or your personal project but cloud services can help you. Create a free account on Azure (you get 200$ worth free credit for 1 month) and the account is free for1 year.

You can run your project there using Azure Machine Learning/AutoML/Python SDK whatever you choose. Use the free 200$ credit in 1 month for any kind of large scale project which requires large computational power or large memory requirement. The link will give you an idea about the computational power and memory capacities on Azure.

$\endgroup$
2
1
$\begingroup$

Are many of the entries in the matrix zero? In this case, you can often deal with large matrices without using large amounts of memory. Sparse matrix data structures exist and so do algorithms for doing arithmetic with them.

scipy includes support for sparse matrices. https://docs.scipy.org/doc/scipy/reference/sparse.html

$\endgroup$
1
$\begingroup$

You can use Dask array, in which you can create essentially any size array and apply some transformation. What it basically does is load a chunk of the array that can be fit into memory process it and then save it. Check the following link :

[https://examples.dask.org/array.html][1]

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.