0
$\begingroup$

I would like to cluster some user reviews and I'm doing this with k-means. In my dataset I have English and German reviews. Is this manipulating the cluster result if I don't seperate them? Or should I do a k-means prediction on each language?

$\endgroup$
2
$\begingroup$

Yes you should definitely split these two up. When calculating the TF-TDF matix, it will give different terms to objects of the same entity, because of the difference in language. This will affect your clustering results.

$\endgroup$
0
$\begingroup$

Using text documents in different languages you are going to have different vector representations, unless you translate the documents previously. For example, house and maison are going to be related to different features. So a cluster algorithm is not going to recognize them as synonymous.

You should try a previous translation of your reviews. The quality of that translation is going to affect the clustering algorithm depending on the algorithms you are using.

If you tell me the steps you are performing in your cluster I could help you better.

$\endgroup$
  • $\begingroup$ My cluster is right now pretty simple. I load the Reviews, then i vectorize them with TfidfVectorizer() and during that, i clean them (delete stop words and punctution and lower the letters. Next I start the training $\endgroup$ – Nika Apr 11 '18 at 20:37
  • $\begingroup$ Ok, that could work but your results are going to be clusterized by language. tfidf has not way of realizing which words are translations between languages. You should perform a translation before you remove the stop words, or after that, if you have stop words in different languages. You can try googletrans to translate your reviews. $\endgroup$ – Federico Caccia Apr 11 '18 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.