Can you l2 normalize word2vec vectors for density clustering?

Question

I have a situation where I have to cluster word2vec vectors (200 length dimension vectors on a very large corpus). I decided to use Density based clustering (DBSCAN, HDBSCAN) because my dataset is very high in noise, and I do not want it to be part of my clusters. My knowledge on Cosine distance is limited but I find that Density based clustering algorithms do not have a direct implementation using cosine distance (pairwise_distance calculation is too memory intensive).

My question here is can I normalize the Word2vec vectors using L2 normalization using: norm_data = normalize(vector_array, norm='l2') from the sklearn library to normalize these vectors, and then use euclidean distance over the normalized vectors?

Can someone suggest any other better technique to cluster word vectors when there is noise in the dataset?

Answer 1

Euclidean distances between l2-normalized word vectors is equivalent to the angular distance between the word vectors. This is not the same as cosine distance, but it is very similar. It is worth noting that cosine distance is not actually a metric, and that forces many algorithms, including the ones discussed here, to resort to more expensive O(n^2) asymptotic performance.

With that in mind I have done quite well clustering similar dimension word vectors trained with word2vec using HDBSCAN using exactly such an l2-normalizing approach as you suggest here. In general I did find it beneficial to use the leaf cluster extraction method rather than the traditional excess of mass approach to get tight clusters. I found the results to be quite good, with clusters providing very clean "topics" for the corpus on which word2vec was trained.

Answer 2

1. You can do DBSCAN, OPTICS, HDBSCAN with cosine similarity - they do not expect or require Euclidean distances.

2. Yes, one version of cosine distance corresponds nicely to Euclidean distance of L2 normalized vectors. So there is some theoretical support to L2 normalize the vectors and try Euclidean then.

3. Given how word2vec is trained, I don't think Euclidean nor cosine distance is the right thing to do. Instead, use the raw dot product as a similarity measure (beware that it is not limited to [0;1] but allows negative similarity - values may be too spread out to use them intuitively).

4. t-SNE, which was discussed in the comments and another answer, also relies on distances to the nearest neighbors. So it does suffer from the same problem - cosine, Euclidean, dot product? The fast ELKI implementation does support cosine distance as well as Euclidean, but you appear to be set on Python already. Also, if you have a large vocabulary, t-SNE may give you some scalability trouble. The standard algorithm will need O(n²) memory and time.

Answer 3

Normalizing the vectors maps them all down to a unit sphere. That definitely changes their Euclidean distances (not cosine distances though), so, no I don't think that's valid, nor does it help the dimensionality.

Clustering in high dimensions is indeed problematic and I agree with @Emre that t-SNE is an option to get more meaningful clusters out for visualization purposes. Simple PCA followed by clustering might be effective too.