0
$\begingroup$

I had a post on stackoverflow. As it is related to scikit-learn, I am hoping that I can obtain some assistance from data scientists in this forum. https://stackoverflow.com/questions/38640815/python-running-out-of-memory

In general, however, I would like to seek opinions on the reduction of memory use when one uses scikit-learn, as this can be a daily problem to deal with as a data scientist. Thank you very much.

$\endgroup$
1
$\begingroup$

I don't know if it is still an open question, anyway.

What is painful when using TFidf, it is the size of the ouput sparse matrix. II think it is your problem here. In order to pass this, you should limit the amount of word you want to treat. I didn't try with your case but you should try CountVectorizer before TfidfTransformer or directly TfidfVectorizer. In CountVectorizer and TfidfVectorizer from sklearn, you can specify the minimum and maximum number of document any term can be found in (min_df and max_df). This will reduce the size of the output A.

Hope it helps a bit

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.