8

When to use cosine similarity over Euclidean similarity Cosine similarity looks at the angle between two vectors, euclidian similarity at the distance between two points. Let's say you are in an e-commerce setting and you want to compare users for product recommendations: User 1 bought 1x eggs, 1x flour and 1x sugar. User 2 bought 100x eggs, 100x flour ...


8

You cannot really use k-means clustering if your data contains categorical variables since k-means uses Euclidian distance which will not make a lot of sense with categorical variables. Check out the answers to this similar question. You can use the following rules for performing clustering with k-means or one of its derivates: If your data contains only ...


5

The are some techniques to choose the number of clusters K. The most common ones are The Elbow Method and The Silhouette Method. Elbow Method In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion. Then, you plot them and where the function creates "...


4

Some unsupervised models can make predictions, but not ones that necessarily match the original class labels. Once a GaussianMixture model has been fitted, it can predict which of the clusters a new example belongs to. This is exactly what the predict and predict_proba functions do in this case, and given that the number of clusters is set to 3, the number ...


4

There is no difference in methodology between 2 and 4 columns. If you have issues then they are probably due to the contents of your columns. K-Means wants numerical columns, with no null/infinite values and avoid categorical data. Here I do it with 4 numerical features: import pandas as pd from sklearn.datasets.samples_generator import make_blobs from ...


4

In my opinion there are two ways: Ask a few experts to assess the quality of the clusters based on a sample (after the clustering has been done, much easier than pre-annotating the whole data especially in the case of clustering) If the clustering is done in the perspective of using the result in another task, the performance of this other task will reflect ...


4

They won't necessarily be the same. Consider observations equally distributed over a circle (radius = 1). Depending on the initial centroids, the algorithm will converge on different solutions. For instance, consider the case where two centroids are initially located on each side of one of the circle's diameters. Those can be any pair of points, and the ...


4

You have two options: 1) Let the K-means algorithm run for a large number of iterations (if on sklearn, change the max_iter parameter value for sklearn.cluster.KMeans). It will eventually converge to a good result (but it will take more time) 2) Make and "educated guess" for the initial starting point. One way to do that is to transform your data in a ...


4

Neither, there is not enough discriminatory information in data (yet) Dont squeeze the data until it tells you the truth. You can change the metric (malahobian distance for example) and the algo but you cant expect it to show miracles. Using elbow method, as you increase the number of clusters it will always become more homogenous. You dont have a "kink" ...


3

bert-as-service (https://github.com/hanxiao/bert-as-service#building-a-qa-semantic-search-engine-in-3-minutes) offers just that solution. To answer your question, implementing it yourself from zero would be quite hard as BERT is not a trivial NN, but with this solution you can just plug it in into your algo that uses sentence similarity.


3

Fuzzy C-means is implemented in Python and you just need to google it e.g. this one, however you can implement it yourself as well. My answer will be more about your task. You have categorical data which means any data point in your problem is on the corner of a high-dimensional simplex. A simple example is 3 points on three vertices of a triangle. How ...


3

Generally, clustering on separate categorical and numerical features is wrong since it could lead to merging the otherwise separate clusters. Here is a visual example of why this may fail (drawn by myself): If we cluster only on the categorical feature, clusters C1 and C2 would be merged. If we cluster only on the numerical feature, all three clusters would ...


3

There is nothing wrong with not using all attributes. In fact there are subspace clustering approaches that attempt to identify (partially) informative attributes along with clusters (but mostly for continuous variables). On your data, you will have big data preparation issues, that would need careful weighting and nonlinear transformations. So it probably ...


3

Clearly the objective function uses a sum over the features. So if you want to increase the importance of a feature, scale it accordingly. If you scale it by 2, the squares grow by 4. So you have increased the weight. However, I would just not use k-means for one-hot variables. The mean is for continuous variables, minimizing the sum of squares on a one-...


3

You cannot use k-means clustering algorithm, if your data contains categorical variables and k-modes is suitable for clustering categorigal data. However, there are several algorithms for clustering mixed data, which actually are variations\modifications of the basic ones. Please check the following paper: "Survey of State-of-the-Art Mixed Data Clustering ...


3

The complexity of that algorithm is O(n³), and it needs O(n²) memory. So if your data grows "exponentially", you better settle for a sampling-based approach! Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for ...


3

Two ideas come to mind, which could be combined or not. Try to identify the single point as an outlier, and remove it from consideration for the clustering. Allow $k$ to vary a little. Using both and allowing $k\in\{2,3\}$ allows you to find only two groups in the main set of points, plus the outlier. Using just (2) with $k\in\{3,4\}$ could find clusters ...


3

When your data has outliers K-means is not a good choice (as you can understand from the way the algorithm behave). In your case I suggest using k-medoids (also called PAM for Partitioning Around Medoids) which isn't outlier-sensitive. K-medoids takes more time to compute compared to K-means but you'll not notice the difference with such order of ...


3

You want a supervised approach. Clustering will not care about your target variable and perform arbitrary splits that don't help. Likely a decision tree can be helpful here, if you use a good implementation that can split the same feature multiple times (to break it into intervals). There are other approaches you could try, such as piecewise regression etc.


3

Always use a supervised algorithm when you have labeled data for your problem. Why would you ignore the labels, your most valuable bit of information? To improve quality, you most likely need to improve your features.


3

Looking at your different steps, the important thing to do is check which step would be affected by outliers. Removing missing values is not affected because this step is not dependent on other data points present (or not) in the dataset. However, normalizing your data is. Indeed, let's say your outliers contain extreme values, this will affect the ...


3

K-Medoids It would be possible with an adapted semi-supervised K-Means, also known as K-Medoids. The tricky part with K-Means is that you do not know the centroids. However, you could hot start by assuming that some of your data points are centroids. Then, when figuring the new centroid at each iteration, instead of figuring out the "imaginary" central ...


2

This is mostly an issue with really bad initialization (random vector generation as well as random labeling are stupid, don't use it - choose k points wth sampling, or k-means++) and with data where k-means doesn't work well at all. So if this happens, you know the results won't be good! Either way, the standard and straightforward solution is simple: use ...


2

If you get an empty cluster, it has no center of mass. You can simply ignore this cluster (set k=k-1 for next iteration), or repeat the k-means run from a new initialization. You can also choose to place a random data point into that cluster and carry on with the algorithm if you must have this specific number of K clusters. If it keeps happening, there is ...


2

In short, KMeans is a distance based clustering technique where depending on the distance between the data points your initialization(usually kmeans++) and clustering works. In kmeans, you initialize cluster centers and then find distance between each point and each of the cluster and then you cluster points to their nearest centers. Here the optimization ...


2

You should check out https://github.com/seatgeek/fuzzywuzzy#usage. fuzzywuzzy is an awesome library for string/text matching that gives a number between 0 to 100 based on how similar two sentences are. It uses Levenshtein Distance to calculate the differences between sequences in a simple-to-use package. Also, check out this blog post for a detailed ...


2

Within the documentation for HDBSCAN (Hierarchical DBSCAN), there is a really nice comparison of clustering algorithms. It is a bit biased, highlighting its own strengths (of course), but will still give you the examples and some boilerplate code to get up and running quickly. DBSCAN and HDBSCAN are generally known not to be so good at handling high variance ...


2

There are hundreds of algorithms to choose from. Hierarchical clustering in it's myriad of variants. Cut the dendrogram as desired, e.g., to get k clusters PAM, the closest match to k-means on a distance matrix (minimizes the average distance from the cluster center) Spectral clustering DBSCAN OPTICS HDBSCAN* Affinity Propagation ...


2

Based on the comments I'll try to answer. I guess you don't have the corresponding labeles. What you can do as a solution is that you can use k-means algorithm as the easiest start point to and specify the hyper parameter, k, to two. Then you can find two clusters and you yourself can evaluate the results. As another approach, you can increase the size of k ...


2

Well, one may argue that DBSCAN is based on all pairwise distances, but it uses data indexing to avoid computing all of them using geometric bounds. And there are other examples if you browse through literature. For example, the classic CLARA method is an approximation to PAM that avoids computing all pairwise distances. And there are many more such ...


Only top voted, non community-wiki answers of a minimum length are eligible