0
$\begingroup$

I have a dataset of over 67 000 records, and I'm trying to run a k-means cluster analysis on that. Orange returns a memory error. The data is in an excel file, but I also tried to load it from a csv. Omitting some records and downsizing it under 65k records also couldnt help. Hierarchical clustering also doesnt work for the same reason.

Does anyone have any ideas on what to do here?

Error code:

MemoryError
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2220, in b.button = btn = button(b, master, label, callback=lambda: do_commit()) File "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2191, in do_commit commit()
File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py", line 246, in run self.run_optimization()
File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py", line 225, in run_optimization self.optimization_runs.append((k, kmeans(self.data)))
File "C:\Python34\lib\site-packages\Orange\projection\base.py", line 28, in call clf = self.fit(data.X, data.Y)
File "C:\Python34\lib\site-packages\Orange\clustering\kmeans.py", line 23, in fit proj = proj.fit(X, Y)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 821, in fit n_jobs=self.n_jobs)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 277, in k_means tol = _tolerance(X, tol)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 162, in _tolerance variances = np.var(X, axis=0)
File "C:\Python34\lib\site-packages\numpy\core\fromnumeric.py", line 2938, in var keepdims=keepdims)
File "C:\Python34\lib\site-packages\numpy\core_methods.py", line 102, in _var x = asanyarray(arr - arrmean) MemoryError

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you please provide the error and the code you are using to try to load the csv file? $\endgroup$
    – Brandon Loudermilk
    Jun 15, 2016 at 12:14
  • $\begingroup$ just added that to the original post. Thanks! With regards to loading the CSV, I'm using the GUI, havent written any codes to do that. $\endgroup$
    – tkazmer
    Jun 16, 2016 at 6:43
  • 1
    $\begingroup$ You said you tried downsampling from 67k to 65k. Try to be more drastic and downsample to 30k or even less. You need to do some debugging like that so that we can narrow down the bug. $\endgroup$
    – Ricardo Magalhães Cruz
    Jun 16, 2016 at 8:56
  • 1
    $\begingroup$ 67k records of how many features, i.e. how wide? $\endgroup$
    – K3---rnc
    Jun 30, 2016 at 16:47

1 Answer 1

Reset to default
1
$\begingroup$

Only 32-bit Orange is currently available for Windows (the filename says win32, if you look). So your per-process RAM is limited to 4 GB. The exception says you ran out of RAM.

Use 64-bit Anaconda installer to get 64-bit Python and libraries on Windows, and then install Orange 3 there.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.