1
$\begingroup$

I have a situation where I have to cluster word2vec vectors (200 length dimension vectors on a very large corpus). I decided to use Density based clustering (DBSCAN, HDBSCAN) because my dataset is very high in noise, and I do not want it to be part of my clusters. My knowledge on Cosine distance is limited but I find that Density based clustering algorithms do not have a direct implementation using cosine distance (pairwise_distance calculation is too memory intensive).

My question here is can I normalize the Word2vec vectors using L2 normalization using: norm_data = normalize(vector_array, norm='l2') from the sklearn library to normalize these vectors, and then use euclidean distance over the normalized vectors?

Can someone suggest any other better technique to cluster word vectors when there is noise in the dataset?

$\endgroup$
  • $\begingroup$ Clustering is problematic in such high dimensions. For visualization purposes I'd use t-SNE on a 3D projection. If the goal is not visualization specify it and maybe we can suggest a workaround. $\endgroup$ – Emre Jul 26 '17 at 16:39
  • $\begingroup$ What amount of data you are talking about? Have executed a pipeline of word2Vec, clustering and random forest on text corpus data consisting of 40,000 blogs. Took me less than 20 sec was doing it with Spark ML-Lib, one node cluster. You can explore Spark if performance is the only bottleneck here. $\endgroup$ – vaibhavbarmy Jul 27 '17 at 9:43
3
$\begingroup$
  1. You can do DBSCAN, OPTICS, HDBSCAN with cosine similarity - they do not expect or require Euclidean distances.

  2. Yes, one version of cosine distance corresponds nicely to Euclidean distance of L2 normalized vectors. So there is some theoretical support to L2 normalize the vectors and try Euclidean then.

  3. Given how word2vec is trained, I don't think Euclidean nor cosine distance is the right thing to do. Instead, use the raw dot product as a similarity measure (beware that it is not limited to [0;1] but allows negative similarity - values may be too spread out to use them intuitively).

  4. t-SNE, which was discussed in the comments and another answer, also relies on distances to the nearest neighbors. So it does suffer from the same problem - cosine, Euclidean, dot product? The fast ELKI implementation does support cosine distance as well as Euclidean, but you appear to be set on Python already. Also, if you have a large vocabulary, t-SNE may give you some scalability trouble. The standard algorithm will need O(n²) memory and time.

| improve this answer | |
$\endgroup$
3
$\begingroup$

Euclidean distances between l2-normalized word vectors is equivalent to the angular distance between the word vectors. This is not the same as cosine distance, but it is very similar. It is worth noting that cosine distance is not actually a metric, and that forces many algorithms, including the ones discussed here, to resort to more expensive O(n^2) asymptotic performance.

With that in mind I have done quite well clustering similar dimension word vectors trained with word2vec using HDBSCAN using exactly such an l2-normalizing approach as you suggest here. In general I did find it beneficial to use the leaf cluster extraction method rather than the traditional excess of mass approach to get tight clusters. I found the results to be quite good, with clusters providing very clean "topics" for the corpus on which word2vec was trained.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Normalizing the vectors maps them all down to a unit sphere. That definitely changes their Euclidean distances (not cosine distances though), so, no I don't think that's valid, nor does it help the dimensionality.

Clustering in high dimensions is indeed problematic and I agree with @Emre that t-SNE is an option to get more meaningful clusters out for visualization purposes. Simple PCA followed by clustering might be effective too.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You assume that the Euclidean distances are worth preserving, but it seems to be more common to use cosine with word2vec. Also, t-SNE will require a good distance metric on the input data, so it suffers from the same problem. Usually it's applied to Euclidean distances, so the discussed approach may improve t-SNE results unless you use e.g. ELKIs t-SNE which supports cosine. $\endgroup$ – Has QUIT--Anony-Mousse Aug 3 '17 at 7:02
  • $\begingroup$ The OP says he uses Euclidean distance. $\endgroup$ – Sean Owen Aug 3 '17 at 11:10
  • $\begingroup$ Not quite, he suggest to use L2 normalization and Euclidean to approximate cosine. $\endgroup$ – Has QUIT--Anony-Mousse Aug 6 '17 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.