127

The standard k-means algorithm isn't directly applicable to categorical data, for various reasons. The sample space for categorical data is discrete, and doesn't have a natural origin. A Euclidean distance function on such a space isn't really meaningful. As someone put it, "The fact a snake possesses neither wheels nor legs allows us to say nothing about ...


52

K-means is not the most appropriate algorithm here. The reason is that k-means is designed to minimize variance. This is, of course, appearling from a statistical and signal procssing point of view, but your data is not "linear". Since your data is in latitude, longitude format, you should use an algorithm that can handle arbitrary distance functions, in ...


36

Your choices of activation='softmax' in the last layer and compile choice of loss='categorical_crossentropy' are good for a model to predict multiple mutually-exclusive classes. Regarding more general choices, there is rarely a "right" way to construct the architecture. Instead that should be something you test with different meta-params (such as layer ...


24

In my opinion, there are solutions to deal with categorical data in clustering. R comes with a specific distance for categorical data. This distance is called Gower (http://www.rdocumentation.org/packages/StatMatch/versions/1.2.0/topics/gower.dist) and it works pretty well.


22

Well the names are pretty straight-forward and should give you a clear idea of vector representations. The Word2Vec Algorithm builds distributed semantic representation of words. There are two main approaches to training, Distributed Bag of Words and The skip gram model. One involves predicting the context words using a centre word, while the other ...


21

For clustering, your data must be indeed integers. Moreover, since k-means is using euclidean distance, having categorical column is not a good idea. Therefore you should also encode the column timeOfDay into three dummy variables. Lastly, don't forget to standardize your data. This might be not important in your case, but in general, you risk that the ...


20

(In addition to the excellent answer by Tim Goodman) The choice of k-modes is definitely the way to go for stability of the clustering algorithm used. The clustering algorithm is free to choose any distance metric / similarity score. Euclidean is the most popular. But any other metric can be used that scales according to the data distribution in each ...


20

Normalization is not always required, but it rarely hurts. Some examples: K-means: K-means clustering is "isotropic" in all directions of space and therefore tends to produce more or less round (rather than elongated) clusters. In this situation leaving variances unequal is equivalent to putting more weight on variables with smaller variance. ...


18

First of all, sklearn.metrics.mutual_info_score implements mutual information for evaluating clustering results, not pure Kullback-Leibler divergence! This is equal to the Kullback-Leibler divergence of the joint distribution with the product distribution of the marginals. KL divergence (and any other such measure) expects the input data to have a sum of ...


17

This question seems really about representation, and not so much about clustering. Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some categorical variable called "color" that could take on the values red, blue, or yellow. If we simply encode these numerically as 1,2, and 3 respectively, our algorithm ...


16

Cosine Similarity for Vector Space could be you answer: http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii/ Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity? "This is a tree", "This is not a tree" If you want to check the semantic meaning of the ...


15

Scipy's entropy function will calculate KL divergence if feed two vectors p and q, each representing a probability distribution. If the two vectors aren't pdfs, it will normalize then first. Mutual information is related to, but not the same as KL Divergence. "This weighted mutual information is a form of weighted KL-Divergence, which is known to take ...


14

Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.


14

After reading your question, I became curious about the topic of time series clustering and dynamic time warping (DTW). So, I have performed a limited search and came up with basic understanding (for me) and the following set of IMHO relevant references (for you). I hope that you'll find this useful, but keep in mind that I have intentionally skipped ...


13

Check the Stanford NLP Group's open source software (http://www-nlp.stanford.edu/software), in particular, Stanford Classifier (http://www-nlp.stanford.edu/software/classifier.shtml). The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in ...


12

An approach that yields more consistent results is K-means++. This approach acknowledges that there is probably a better choice of initial centroid locations than simple random assignment. Specifically, K-means tends to perform better when centroids are seeded in such a way that doesn't clump them together in space. In short, the method is as follows: ...


12

k-means is based on averages. It models clusters using means, and thus the improvement by adding more data is marginal. The error of the average estimation reduces with 1/sqrt(n); so adding more data pays off less and less... Strategies for such large data always revolve around sampling: If you want sublinear runtime, you have to do sampling! In fact, ...


12

Let us briefly talk about a probabilistic generalisation of k-means: the Gaussian Mixture Model (GMM). In k-means, you carry out the following procedure: - specify k centroids, initialising their coordinates randomly - calculate the distance of each data point to each centroid - assign each data point to its nearest centroid - update the coordinates of the ...


11

Online k-means (more commonly known as sequential k-means) and traditional k-means are very similar. The difference is that online k-means allows you to update the model as new data is received. Online k-means should be used when you expect the data to be received one by one (or maybe in chunks). This allows you to update your model as you get more ...


11

Right now, I only have time for a very brief answer, but I'll try to expand on it later on. What you want to do is a clustering, since you want to discover some labels for your data. (As opposed to a classification, where you would have labels for at least some of the data and you would like to label the rest). In order to perform a clustering on your ...


11

Word embeddings are trained by substitutability, not similarity. If you consider a sentence like "This food is unflavored." Then a good substitute word would be "flavored", and the sentence will still be "correct". In many cases, substitutability arises from similarity (crunchy, crispy) but it does also arise from opposites. You may consider "king" and "...


10

I don't think any of the clustering techniques "just" work at such scale. The most scalable supposedly is k-means (just do not use Spark/Mahout, they are really bad) and DBSCAN (there are some good distributed versions available). But you will be facing many other challenges besides scale because clustering is difficult. It's not as if it's just enough to ...


10

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space. It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step. You would map each input vector $x_i$ to a ...


9

I think a number of clustering algorithms that normally use a metric, do not actually rely on the metric properties (other than commutativity, but I think you'd have that here). For example, DBSCAN uses epsilon-neighborhoods around a point; there is nothing in there that specifically says the triangle inequality matters. So you can probably use DBSCAN, ...


9

One possibility here (and this is really an extension of what Sean Owen posted) is to define a "stable user." For the given info you have you can imagine making a user_id that is a hash of ip and some user agent info (pseudo code): uid = MD5Hash(ip + UA.device + UA.model) Then you flag these ids with "stable" or "unstable" based on usage heuristics you ...


8

Alex made a number of good points, though I might have to push back a bit on his implication that DBSCAN is the best clustering algorithm to use here. Depending on your implementation, and whether or not you're using accelerated indices (many implementations do not), your time and space complexity will both be O(n2), which is far from ideal. Personally, my ...


8

The data set definitions are on the page here: Attribute Information at the bottom or you can see inside the ZIP folder the file named activity_labels, that has your column headings inside of it, make sure you read the README carefully, it has some good info in it. You can easily bring in a .csv file in R using the read.csv command. For example if you ...


8

K-means should be right in this case. Since k-means tries to group based solely on euclidean distance between objects you will get back clusters of locations that are close to each other. To find the optimal number of clusters you can try making an 'elbow' plot of the within group sum of square distance. This may be helpful (http://nbviewer.ipython.org/...


8

Please see my comment above and this is my answer according to what I understood from your question: As you correctly stated you do not need Clustering but Segmentation. Indeed you are looking for Change Points in your time series. The answer really depends on the complexity of your data. If the data is as simple as above example you can use the difference ...


8

Taking a stab: I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. Gower Distance is a useful distance metric when the data contains both continuous and categorical variables. There are techniques in R kmodes clustering and kprototype that are designed for this type of ...


Only top voted, non community-wiki answers of a minimum length are eligible