0
$\begingroup$

I have a dataset of over 67 000 records, and I'm trying to run a k-means cluster analysis on that. Orange returns a memory error. The data is in an excel file, but I also tried to load it from a csv. Omitting some records and downsizing it under 65k records also couldnt help. Hierarchical clustering also doesnt work for the same reason.

Does anyone have any ideas on what to do here?

Error code:

MemoryError
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2220, in b.button = btn = button(b, master, label, callback=lambda: do_commit()) File "C:\Python34\lib\site-packages\Orange\widgets\gui.py", line 2191, in do_commit commit()
File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py", line 246, in run self.run_optimization()
File "C:\Python34\lib\site-packages\Orange\widgets\unsupervised\owkmeans.py", line 225, in run_optimization self.optimization_runs.append((k, kmeans(self.data)))
File "C:\Python34\lib\site-packages\Orange\projection\base.py", line 28, in call clf = self.fit(data.X, data.Y)
File "C:\Python34\lib\site-packages\Orange\clustering\kmeans.py", line 23, in fit proj = proj.fit(X, Y)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 821, in fit n_jobs=self.n_jobs)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 277, in k_means tol = _tolerance(X, tol)
File "C:\Python34\lib\site-packages\sklearn\cluster\k_means_.py", line 162, in _tolerance variances = np.var(X, axis=0)
File "C:\Python34\lib\site-packages\numpy\core\fromnumeric.py", line 2938, in var keepdims=keepdims)
File "C:\Python34\lib\site-packages\numpy\core_methods.py", line 102, in _var x = asanyarray(arr - arrmean) MemoryError

$\endgroup$
  • $\begingroup$ Can you please provide the error and the code you are using to try to load the csv file? $\endgroup$ – Brandon Loudermilk Jun 15 '16 at 12:14
  • $\begingroup$ just added that to the original post. Thanks! With regards to loading the CSV, I'm using the GUI, havent written any codes to do that. $\endgroup$ – tkazmer Jun 16 '16 at 6:43
  • 1
    $\begingroup$ You said you tried downsampling from 67k to 65k. Try to be more drastic and downsample to 30k or even less. You need to do some debugging like that so that we can narrow down the bug. $\endgroup$ – Ricardo Cruz Jun 16 '16 at 8:56
  • 1
    $\begingroup$ 67k records of how many features, i.e. how wide? $\endgroup$ – K3---rnc Jun 30 '16 at 16:47
1
$\begingroup$

Only 32-bit Orange is currently available for Windows (the filename says win32, if you look). So your per-process RAM is limited to 4 GB. The exception says you ran out of RAM.

Use 64-bit Anaconda installer to get 64-bit Python and libraries on Windows, and then install Orange 3 there.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.