5
$\begingroup$

I run AgglomerativeClustering on a sample of data and fit a model. then I decide to predict this fit for all of my data but I got MemoryError.

How can I run AgglomerativeClustering on a big dataset? should I create a classification based on clusters label?!

Improve this question
$\endgroup$

3 Answers 3

Reset to default
10
$\begingroup$

You can't.

By definition, the algorithm needs O(n²) memory and O(n³) runtime.

This does not scale to big data.

Use a different algorithm. Or subsample your data.

Results don't necessarily get better just because you use more data. In many cases it really does not matter. The quality of estimating the mean grows with sqrt(n), so it quite quickly does not pay off to use more data, as this will only affect small digits of the result.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ >Results don't necessarily get better just because you use more data. My_Answer: I ran the algorithm on 0.02% of my data and I got the result but the problem raised when I need to label all records. so, my problem is with predicting not fitting!!! $\endgroup$
    – parvij
    Commented Mar 24, 2019 at 21:40
  • $\begingroup$ You can always train a classifier to predict the remaining points consistent with the clustered sample. For example, a nearest neighbor classifier. But if you think you need a label for every point, you are probably using clustering the wrong way. Don't trust the labels. Study the patterns found, but don't rely on them. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Mar 24, 2019 at 22:08
  • $\begingroup$ I do clustering for marketing, I want the label for each point because I want to do some special activities for each cluster. so, I studied clusters, I found them useful but after that, I need to use labels! (thanks Anony) $\endgroup$
    – parvij
    Commented Mar 26, 2019 at 19:14
  • $\begingroup$ Define rules based on your findings to label the points exactly as you want them. Then apply these rules to your data. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Mar 27, 2019 at 0:01
3
$\begingroup$

A sub-sample of records should be perfectly fine if your data is (at least somewhat) normally distributed.

# To get 3 random rows 
# each time it gives 3 different rows 
# df.sample(3) or 
df.sample(n = 3) 


# Fraction of rows 
# here you get .50 % of the rows 
df.sample(frac = 0.5)

Don't burn through hours and hours of your time. Get a small sample working and build on that.

Improve this answer
$\endgroup$
1
$\begingroup$

Counterintuitively, using a precomputed distance matrix (metric="precomputed") reduces runtime.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.