I'm curious if anyone else has run into this. I have a data set with about 350k samples, each with 4k sparse features. The sparse fill rate is about 0.5%. The data is stored in a
scipy.sparse.csr.csr_matrix object, with
I'm using this as an input to sklearn's Logistic Regression classifier. The documentation indicates that sparse CSR matrices are acceptable inputs to this classifier. However, when I train the classifier, I get extremely bad memory performance; the memory usage of my process explodes from ~150 MB to fill all the available memory and then everything grinds to a halt as memory swapping to disk takes over.
Does anyone know why this classifier might expand the sparse matrix to a dense matrix? I'm using the default parameters for the classifier at the moment, within an updated anacoda distribution. Thanks!
scipy.__version__ = '0.14.0' sklearn.__version__ = '0.15.2'