3
$\begingroup$

I'm just starting to work on a relatively large dataset after ML course in Coursera. Trying to work on https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD. Got an accuracy of 5.2 in training and test set with linear regression using gradient descent in octave.

I tried adding all possible quadratic features (515345 instances and 4275 features), but the code just won't stop executing in my HP Pavilion g6 2320tx, with 4GB RAM in Ubuntu 14.04.

Is this beyond the data size capacity of Octave ?

Improve this question
$\endgroup$
3
  • $\begingroup$ How many columns did you get after including the quadratic features? Was it 180? Also, what was the maximum memory usage without including the quadratic term? $\endgroup$
    – Nitesh
    Nov 7, 2014 at 4:52
  • $\begingroup$ I'm trying to add all possible quadratic features. That should give about 4275 features I think. Adding just the squared features works fine. That would be 180. btw how do we check the max memory usage ? $\endgroup$
    – abhivij
    Nov 7, 2014 at 5:08
  • $\begingroup$ see ubuntuforums.org/showthread.php?t=1161120 $\endgroup$
    – Nitesh
    Nov 7, 2014 at 5:55

1 Answer 1

Reset to default
2
$\begingroup$

You have about 4GB of RAM on your machine and Octave is an in memory application.

If you want to work with 515345 instances and 4275 features, assuming that you are using double precision (i.e. 8 bytes), you would need a memory of 515345*4275*8/1000000/1024 bytes ~ 17.6 GB. Even if you were using 4 bytes for each data point, you would require at least 9 GB for the computation to go through.

This issue might not be the Octave memory restriction in this case. See here for further details on Octave's memory usage.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. So adding all quadratic features is not possible. Guess I'll have to try some other methods $\endgroup$
    – abhivij
    Nov 7, 2014 at 9:36
  • $\begingroup$ Yes, that would make more sense. I think 4275 is too many features, given the number of rows you are dealing with. $\endgroup$
    – Nitesh
    Nov 8, 2014 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.