I have two datasets of a similar theme. Let's assume Dataset A and Dataset B. Using the top2vec model (https://github.com/ddangelov/Top2Vec) (https://arxiv.org/abs/2008.09470) on each dataset, I came up with a certain number of topics. Now, I want to compare both datasets' topics. How can I do this? Clustering or any other method will work as far as I can compare the generate topics computationally.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ By comparing do you mean obtaining a single similarity score between the two clustering results? Or something more detailed, like finding the similarity between a topic in dataset 1 and a topic in dataset 2? $\endgroup$– ErwanCommented Dec 27, 2020 at 12:19
-
$\begingroup$ @Erwan Thank you for your interest. Actually, if possible both premises will be very good. If I understand your suggestion correctly (correct me if I wrongly understood) 1) your first suggestion - the similarity score between two clustering results - will provide a broad understanding. 2) your second suggestion - will help in a detailed study. So I think both methods will be a good addition. $\endgroup$– Piyush GhasiyaCommented Dec 28, 2020 at 2:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I'm not aware of any standard options to do that, but I could suggest a couple directions. Note that in both ideas it's possible to compare topics individually or globally.
Applying evaluation measures on the instances
In this idea you would apply the model obtained from dataset A on all the instances of dataset B (and of course you can do the converse as well). This would give you a prediction of topic for every instance in B (typically using the max probability topic for the instance) from both model A and model B.
Then you would need a method to match the topics between A and B, i.e. convert topics from A to B. This can be done using the number of instances they have in common, I know that there is some literature and several methods which have been used in supervised evaluation measures for word sense induction, I would guess that there are similar things for topic modeling (not sure). Once the matching is done, various evaluation measures can be used to compare the topics. The idea is to consider the topics obtained from A as the predictions to be compared against the gold-standard topics from B. Note that most measures are symmetrical, so it doesn't matter which one is considered the prediction or the gold.
Comparing words probabilities
I'm not familiar with top2vec but usually it's possible to obtain the top words for every topic together with their associated probability. Assuming this is possible here, the idea would be to compare either:
- The top N words for every pair of topics (every topic from A against every topic from B). This can be done very simply by counting the number of words in common between the two top N words: the more the topics are similar, the more they have words in common in their top N.
- the whole probability distribution of the words for every pair of topics, using a distance measure such as KL divergence (there are other options). This value will represent how closely related two topics are.
