I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to achieve this?
I don't know if this is your job work or your personal project but cloud services can help you. Create a free account on Azure (you get 200$ worth free credit for 1 month) and the account is free for1 year.
You can run your project there using Azure Machine Learning/AutoML/Python SDK whatever you choose. Use the free 200$ credit in 1 month for any kind of large scale project which requires large computational power or large memory requirement. The link will give you an idea about the computational power and memory capacities on Azure.
Are many of the entries in the matrix zero? In this case, you can often deal with large matrices without using large amounts of memory. Sparse matrix data structures exist and so do algorithms for doing arithmetic with them.
scipy includes support for sparse matrices. https://docs.scipy.org/doc/scipy/reference/sparse.html
You can use Dask array, in which you can create essentially any size array and apply some transformation. What it basically does is load a chunk of the array that can be fit into memory process it and then save it. Check the following link :