NLP topic clustering

Question

In my dataset, I have 500 abstracts. The goal is to cluster them in 2 topics.
One topic must have those abstracts which contain some list of words or similar words and the rest of the abstracts in other topic.
Can anyone kindly offer me suggestions to do this?

Answer 1

for clustering the abstracts, I would suggest the following steps:

1. In order to make the abstracts mathematically comparable, we need to convert these to vector representation. This can be, for example, using a word2vec model to get vector representations for each meaningful word (excluding words such as 'a', 'the', for example) and maybe, for example, taking the average of the vectors to represent a single abstract.
2. Now we have a way to represent abstracts, we can now mathematically compare them. To cluster them, one obvious way to do this a clustering algorithm, such as K-means clustering. In your case, we want to set k (number of clusters) to 2.
   Note: such clustering algorithms are non-deterministic. This means that every time you run the clustering algorithm, you will get different results.
   If you want something more deterministic, then I would recommend something like hierarchical clustering and take the clusters, when it reaches the desired number of clusters (2 in this case).

Answer 2

You can also leverage the combination of transformers and tf-idf for topic clustering. A perfect example of this is BERTopic

pip install bertopic

Then, I would extract the top k topics, where k in your case would be 2.

from bertopic import BERTopic
from sklearn.datasets import fetch_20newsgroups

docs = fetch_20newsgroups(subset='all',  remove=('headers', 'footers', 'quotes'))['data']

topic_model = BERTopic()
topics, probs = topic_model.fit_transform(docs)

Read more on BERTopic here.