2
$\begingroup$

I tried to use OMP algorithm available in scikit-learn. My net datasize which includes both target signal and dictionary ~ 1G. However when I ran the code, it exited with mem-error. The machine has 16G RAM, so I don't think this should have happened. I tried with some logging where the error came and found that the data got loaded completely into numpy arrays. And it was the algorithm itself that caused the error. Can someone help me with this or sugggest more memory efficient algorithm for feature selection, or is subsampling the data my only option. Are there some deterministic good subsampling techniques.

EDIT: Relevant code piece:

n=8;
y=mydata[:,0];
X=mydata[:,[1,2,3,4,5,6,7,8]];
#print y;
#print X;
print "here";
omp = OrthogonalMatchingPursuit(n_nonzero_coefs=5,copy_X = False, normalize=True);
omp.fit(X,y);
coef = omp.coef_;
print omp.coef_;
idx_r, = coef.nonzero();
for id in idx_r:
        print coef[id], vars[id],"\n";

The error I get:

File "/usr/local/lib/python2.7/dist-packages/sklearn/base.py", line 324, in score
return r2_score(y, self.predict(X), sample_weight=sample_weight)
File "/usr/local/lib/python2.7/dist-packages/sklearn/metrics/metrics.py", line 2332, in r2_score
numerator = (weight * (y_true - y_pred) ** 2).sum(dtype=np.float64)
MemoryError
$\endgroup$
  • $\begingroup$ There's not much information here. You should provide more detailed diagnostic information, including how you are invoking OMP and with what settings. $\endgroup$ – Sean Owen Nov 3 '14 at 8:27
  • $\begingroup$ Edited with relevant piece of code and the error as well $\endgroup$ – sshanks Nov 4 '14 at 9:52
  • $\begingroup$ It's not hard to imagine that 16G of data would be too much for 16G of memory. Do you read the data straight into numpy arrays? If not, you're doubling the memory footprint. Can you use sparse matrices? $\endgroup$ – jamesmf Nov 13 '15 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.