I do not know if this is the smartest way to do it, but it seems to work for my study case.
To recap, I performed a top2vec algorithm:
import numpy as np
import pandas as pd
from top2vec import Top2Vec
df = pd.read_excel('Preprocessed2.xlsx')
documents = df_small_small['Translated Cleaned'].to_list()
model = Top2Vec(documents, embedding_model='universal-sentence-encoder', speed='deep-learn')
The algorithm finds 30 topics. However, although coherent, they could be grouped into smaller macrotopic since there are some similarities among different topics.
After having inspected them thanks to a word cloud visualization
topic_sizes, topic_nums = model.get_topic_sizes()
for topic in topic_nums:
model.generate_topic_wordcloud(topic)
and printing closer topics to a specific keyword
topic_words, word_scores, topic_scores, topic_nums = model.search_topics(keywords=["queue"], num_topics=5)
topic_nums
>>> array([ 5, 7, 15, 1, 3], dtype=int64)
I was able to detect which topics could have been grouped together and I manually created a list with the indices of the topic that refer to the same macrotopic.
# for example
macrotopic_1 = [1,7,12,17]
Then I implemented this two dumb functions to get the best words and documents for that best represented the macrotopic
def macrotopic_words(topics_words:np.ndarray, word_scores:np.ndarray, list_of_topics:list):
'''
This function takes in input the output of the top2vec model and a list of integers corresponding to topics'ids and returns two dictionaries containing the most important words and their respective value
Input:
- topic_words: (np.ndarray), an array of arrays containing topic words.
- word_scores: (np.ndarray), an array of arrays containing words scores. These scores represent the cosine similarity to each keyword that defines the topic.
- list_of_topics: (list), a list of integers. These integers have to be between 0 and the max number of topics spotted by the top2vec algorithm
Output:
_ diz_words: (dict) dictionary whose keys are the top 50 words and values are their associated score. These scores represent the cosine similarity to each keyword that defines the topic.
'''
# The two objects that will be returned
diz_words = dict()
for topic_id in list_of_topics:
topic_words = topics_words[topic_id].tolist()
topic_words_scores = word_scores[topic_id].tolist()
for i in range(len(topic_words)):
word = topic_words[i]
score = topic_words_scores[i]
if word in diz_words:
old_score = diz_words[word]
diz_words[word] = (old_score + score)/2
else:
diz_words[word] = score
# get only the top 50 words
diz_words = dict(sorted(diz_words.items(), key=lambda item: item[1], reverse=True)[:50])
return diz_words
def macrotopic_docs(df, model, list_of_topics:list):
'''
This function takes in input the output of the top2vec model and a list of integers corresponding to topics'ids and returns two dictionaries containing the most important words and their respective value
Input:
- df: (pd.Dataframe) datframe where I there are at least two columns: one with the orginal text (in my case translated reviews) and one with the pre-processed text (in my case translated reviews).
This is not necessary but for my application makes the result more human readable.
- model: (top2vec.Top2Vec.Top2Vec) output of the top2vec model
- list_of_topics: (list), a list of integers. These integers have to be between 0 and the max number of topics spotted by the top2vec algorithm
Output:
_ diz_docs: (dict) dictionary whose keys are the top 50 words and values are their associated score. These scores represent the cosine similarity to each keyword that defines the topic.
'''
# The two objects that will be returned
diz_words = dict()
for topic_id in list_of_topics:
documents, document_scores, document_ids = model.search_documents_by_topic(topic_num=topic_id, num_docs=10)
for i in range(len(documents)):
doc = documents[i]
doc_nice_idx = df.index[df['Translated Cleaned'] == doc][0]
doc_nice = df.iloc[doc_nice_idx]['Translated reviews']
score = document_scores[i]
if doc_nice in diz_words:
old_score = diz_words[doc_nice]
diz_words[doc_nice] = (old_score + score)/2
else:
diz_words[doc_nice] = score
# get only the top 50 words
diz_words = dict(sorted(diz_words.items(), key=lambda item: item[1], reverse=True)[:10])
return diz_words
macrotopic_words(topic_words,word_scores,[5,7,8] )
>>> {'ticket': 0.3825201094150543,
'queue': 0.2912799119949341,
'long': 0.2060506008565426,
'close': 0.2041976973414421,
'min': 0.19961485266685486,
'hour': 0.18805542960762978,
'stop': 0.1846560537815094,
'near': 0.1766391098499298,
'longer': 0.17402754724025726,
'forget': 0.1692977249622345,
'open': 0.16631683707237244,
'interest': 0.16506552696228027,
'quit': 0.16504628770053387,
'finish': 0.16126474738121033,
'tail': 0.16030896827578545,
'end': 0.15837934613227844,
'much': 0.15592673420906067,
'start': 0.15537860989570618,
'stay': 0.15466280281543732,
'wait': 0.15337903797626495,
'visit': 0.1528526358306408,
'time': 0.15138189494609833,
'work': 0.14888891577720642,
'gate': 0.14842548966407776,
'worth': 0.1457190401852131,
...
'full': 0.13024736940860748,
'second': 0.13011334836483002,
'buy': 0.1292862892150879,
'moment': 0.12763935327529907,
'opportun': 0.1272534765303135}
macrotopic_docs(df, model, [1])
>>> {'On October 3, the Pergamon Museum reopened its gates.The pergamon altar is not yet available.But even that\'s how you can spend interesting hours in the old Orient.Highlights such as the Ischtar goal, the Milet or Diemschatta facade, are accessible.Very good audio guide with pleasantly compact explanations.It was not so nice that you are currently running "backwards" through the exhibition and the audio guide is not adapted to this.This leads to the fact that you can hear the introduction to your specific exhibition area just before you leave this room ... In the introduction, the audio guide also indicates that the pergamon altar is expected to be restored until 2019!In the months of the closure, one could really have bothered to update the audio guide! ...': 0.7808947,
"The Pergamon Museum is an obligatory stop in Berlin.Ishtar's door alone is worth the ticket, but don't stop there.The whole exhibition is absolutely enviable, especially if you take the free audio guide at the entrance, which tells you anecdotes and the story of Pergamo. ...": 0.7628059,
'Beautiful experience. Babilonia Porta magnificent as well as the entry to the Miletus market. Make up the audio guides included in the price of our ticket (Euro 19 on Get Your Guide, with a priority entrance), which has a very good value for moneywhich also provides ...': 0.75661397,
'If anyone goes to Berlin, he must visit this place.Huge impressions !!!Personally, I think that you should ensure a lot of time to visit.The museum delights !!!In addition, at the entrance we get an audio guide and we decide how much time we want to spend on ...': 0.74968064,
"I've never seen anything like that, this museum is unforgettable!The audio guide helps a lot, it's worth it!And it was the first time I saw a hygiene being done in audio guides.They clean each one and give you a cover to the headphones.wonderful!Hot Tip: Buying online is cheaper and cut…": 0.74947065,
"I have already visited this museum twice;It is wonderful, there are entire parts of temples, a piece of the Babylon gate, it is really great.The audio guides allow a guided tour according to the visitor's time and interests.": 0.7454577,
'Unmissable museum if passing through Berlin.Unfortunately, the Pergamo section was not open, but the Babylon Palace and the Mileto market door are certainly worth the trip.Excellent audio guides provided free of charge with the ticket.It is preferable to do the ...': 0.74440473,
"The pergame is wonderful.Everyone who visits Berlin must pass by.The entrance costs € 12 and if you don't have much time, in two hours you can travel it.Free guide and wardrobe audio.Keep in mind that a part of the museum is closed and its great star, the ...": 0.7332305,
'I saw it very willingly because it really represents a piece of history in a unique way.Both the Temple of Pergamon, and the Babylon gate manage to leave you without a word.The audio guide, included in the price of the entrance ticket, is very well done and accompanies ...': 0.7329705,
'I visited Berlin with the family and made a ticket that allowed us to enter all the museums.We made a rather quick visit to this museum but the emotion in seeing the area dedicated to the Ishtar gate was great.Very beautiful the audio guide who ...': 0.7325742}
```
