1
$\begingroup$

I am clustering text based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can spread very randomly, I found Kernel Density Estimators relevant.

However, the scores of KDE are very sensitive to the choice of bandwidth hyper-parameter, which I could not pre-estimate. Most bandwidth values end up with either infinite score for points outside the cluster, of zero for the points in the cluster. I need a way to 'automatically' choose the bandwidth, so that my results will yield scores that make sense for the points in the cluster (larger values) and the points outside of it (smaller values).

I tried:

  • Both Silverman and Scott factors methods to evaluate the bandwidth depending on #points and #features, both were far from relevant in my case
  • GridSearchCV returns the minimal bandwidth in the grid
  • Different kernel types (all the relevant ones are similarly sensitive)
  • Reduce dimension, but as expected this severely hurt the KDE results without making the bandwidth that less sensitive
from sklearn.neighbors import KernelDensity

# indexes of points in cluster3
docs = np.where(y_pred == 3)[0] 

kde = KernelDensity(kernel='gaussian',bandwidth=0.399).fit(X_tfidf[docs].todense())

# evaluate scores on all points
scores = np.exp(kde.score_samples(X_tfidf.todense()))

Note that there are ~2200 features in the TF-IDF, a few dozen points (40-120) in each cluster the KDE was fitted to, and about 4000 points in total.

Any ideas on anything (even beyond KDE) are welcome, thank you.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
0
$\begingroup$

GridSearchCV returns the minimal bandwidth in the grid

Then alter your grid to have a lower lower bound.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The bandwidth it returns makes no sense. For instance, if I found that a bandwidth of about 0.399 makes sense, and bandwidth below 0.38 return infinite scores, then Grid Search would return a bandwidth of the lowest lower bound (even when it's 0.1 or anything below 0.399) $\endgroup$
    – Adam
    Jan 22, 2018 at 11:14
  • $\begingroup$ KDE shouldn't return infinite scores. Might be time for you to show us some code, or better yet a minimal working example. $\endgroup$
    – David Marx
    Jan 22, 2018 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.