4
$\begingroup$

I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a scipy.sparse.csr.csr_matrix object, with dtype='numpy.float64'.

I'm using this as an input to sklearn's Logistic Regression classifier. The documentation indicates that sparse CSR matrices are acceptable inputs to this classifier. However, when I train the classifier, I get extremely bad memory performance; the memory usage of my process explodes from ~150 MB to fill all the available memory and then everything grinds to a halt as memory swapping to disk takes over.

Does anyone know why this classifier might expand the sparse matrix to a dense matrix? I'm using the default parameters for the classifier at the moment, within an updated anacoda distribution. Thanks!

scipy.__version__ = '0.14.0'
sklearn.__version__ = '0.15.2'
$\endgroup$
5
$\begingroup$

Ok, this ended up being an RTFM situation, although in this case it was RTF error message.

While running this, I kept getting the following error:

DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().

I assumed that, since this had to do with the target vector, and since it was a warning only, that it would just silently change my target vector to 1-D.

However, when I explicitly converted my target vector to 1-D, my memory problems went away. Apparently having the target vector in an incorrect form caused it to convert my input vectors into dense vectors from sparse vectors.

Lesson learned: follow the recommendations when sklearn 'suggests' you do something.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.