1
$\begingroup$

I have a pre trained Glove word embedding matrix (U) of dimension (400000 * 50). Now for the purpose of query expansion I need to perform the operation matmul(U*U.T). This is the term by term similarity matrix. But this leads to a matrix size of (400000*400000) and Python is throwing me memoryError.

Is there a work around to this problem other than using a better machine ?

$\endgroup$
1
  • $\begingroup$ Maybe Matrix Factorization? If you Google you will find a lot about it. $\endgroup$ Commented Aug 18, 2018 at 11:18

1 Answer 1

1
$\begingroup$

There are many options. Here are several:

  • Use NumPy, instead of pure Python. Make sure to compile NumPy with an optimized BLAS library.
  • Use a framework like Dask to read only a part of the matrix at a time from disk.
  • Choose a different method query expansion that does not require the inverse.
  • Use the pseudoinverse.
  • Choose a smaller embedding space.
  • Reduce the precision of the numerical values.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.