0
$\begingroup$

I have texts collected on different topics. I would like to study the possible correlation between these. I have started looking at the word frequency and it seems that in one dataset the word with highest frequency is cat; in another dataset is mice; in another one is house.

Do you know about some technique or approach (similarity, classification,...) to show the possible correlation between datasets (e.g. between cat and mice)? Should it need only the expert judgement?

$\endgroup$
0
$\begingroup$

One method you can try first is cosine similarity. It works by counting the number of occurrences of each word in the vocabulary for each individual document. Next, you put these counts into vectors and then you take the cosine of the angle between them.

If you have more than one text on a topic, you can combine them into a single text for the purpose of finding the cosine similarity between various topics.

If you have many texts that are examples of each topic, then you can create a supervised machine learning model. However, since your purpose is interpretation instead of prediction, I would recommend a technique like cosine similarity before getting insight from interpreting a predictive model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thank you so much @thomaskolasa. Yes, I would like to try to do something like count the words frequency within each topic, then try to see if there is correlation. We know that cat and mice are correlated in some way(they are both animal; the cat eats mice;...). Should it work with cosine similarity? It is not clear to me how I could get this information from it. Could you provide me an example, please? $\endgroup$ – Luca Di Mauro May 16 at 19:00
  • $\begingroup$ That would not show up with cosine similarity. Look up Word2vec that automatically looks at word context. $\endgroup$ – thomaskolasa May 18 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.