I want to perform agglomerative clustering, but I have no idea of number of clusters before hand. But I want that every cluster has at least 40 data points in it. How can I apply this to sklearn.agglomerative clustering? Should I use dendrogram and cut it somehow? I have no idea how to relate dendrogram to this and cutting it out. Any help will be appreciated!
A minimum cluster size will not generally be satisfiable in hierarchical clustering. Instead, you have to expect many clusters with just a single point.
ELKI has some fairly interesting techniques to cut a dendrogram. Check the
clustering.hierarchical.extraction (or so) package. If I remember correctly, some allow you to set a minimum size (but there will be a "noise" cluster with all the leftovers).
If you don t know the number of clusters, i encourage you to look at those density based algorithm : Mean Shift, DBSCAN, OPTICS. They don t presume of the cluster number and are able to find random shape clusters.