0
$\begingroup$

In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics.
One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in other topic.
Can anyone kindly offer me suggestions to do this?

$\endgroup$
1
  • $\begingroup$ Cosine similarity. $\endgroup$ Jan 24, 2023 at 6:55

2 Answers 2

0
$\begingroup$

for clustering the abstracts, I would suggest the following steps:

  1. In order to make the abstracts mathematically comparable, we need to convert these to vector representation. This can be, for example, using a word2vec model to get vector representations for each meaningful word (excluding words such as 'a', 'the', for example) and maybe, for example, taking the average of the vectors to represent a single abstract.
  2. Now we have a way to represent abstracts, we can now mathematically compare them. To cluster them, one obvious way to do this a clustering algorithm, such as K-means clustering. In your case, we want to set k (number of clusters) to 2.
    Note: such clustering algorithms are non-deterministic. This means that every time you run the clustering algorithm, you will get different results.
    If you want something more deterministic, then I would recommend something like hierarchical clustering and take the clusters, when it reaches the desired number of clusters (2 in this case).
$\endgroup$
1
$\begingroup$

You can also leverage the combination of transformers and tf-idf for topic clustering. A perfect example of this is BERTopic

pip install bertopic

Then, I would extract the top k topics, where k in your case would be 2.

from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

docs = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))['data']

topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)

Read more on BERTopic here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.