# Thematic clustering of text

Please advise on starting points, research (papers,frameworks) related to thematic clustering of text. In particular on a system with two levels of clustering where second level has a temporal nature. Thanks!

Update:

Sorry for ambiguity in my initial question. I need to clarify, that I have experience with clustering in general and document clustering in particular, for text using TFIDF, word embeddings (word2vec, Glove and BERT Sentence embeddings) in a vector space.

My question originates from some text mentioning "two levels of thematic clustering of text where second level has a temporal nature". Just wanted to know about this technique and in particular "temporal nature" used in clustering.

Your question is way to vague. What did you try, what do you want to achieve in detail. However, find below some R examples how to approach topic modeling. This book might also be helpful: Text Mining with R.

library(topicmodels)
data("AssociatedPress")
AssociatedPress

# LDA (Latent Dirichlet Allocation)
ap_lda <- LDA(AssociatedPress, k = 2, control = list(seed = 1234))
ap_lda

# Get "topics"
library(tidytext)
ap_topics <- tidy(ap_lda, matrix = "beta")
ap_topics

library(ggplot2)
library(dplyr)

# Plot topics
ap_top_terms <- ap_topics %>%
group_by(topic) %>%
top_n(10, beta) %>%
ungroup() %>%
arrange(topic, -beta)

ap_top_terms %>%
mutate(term = reorder(term, beta)) %>%
ggplot(aes(term, beta, fill = factor(topic))) +
geom_col(show.legend = FALSE) +
facet_wrap(~ topic, scales = "free") +
coord_flip()

# (words with) Greatest difference between categories
library(tidyr)

mutate(topic = paste0("topic", topic)) %>%
filter(topic1 > .001 | topic2 > .001) %>%
mutate(log_ratio = log2(topic2 / topic1))

list(beta_spread$term) # Select 10 largest and smallest values of "log_ratio") df1 <- subset(beta_spread, beta_spread$$log_ratio<=max(tail(sort(beta_spread$$log_ratio, decreasing = T),10))) df2 <- subset(beta_spread, beta_spread$$log_ratio>=min(tail(sort(beta_spread$$log_ratio, decreasing = F),10))) df = rbind(df1,df2) df # Plot barplot(sort(df$$log_ratio), names.arg=df$$term[order(df$log_ratio)], las=2)

# Gamma contains the probability of document X to belong to topic Y
ap_documents <- tidy(ap_lda, matrix = "gamma")
ap_documents
# the model estimates that only about 24.8% of the words in document 1 were generated from topic 1

• Thanks! Is thematic clustering the same thing as topic modeling? Where does temporal nature comes into play? Apr 5, 2020 at 20:38
• Please, see my question update. Apr 6, 2020 at 9:00

"Clustering" is a very broad umbrella for a set of unsupervised techniques that tries to group data items together, according to its characteristics/covariates/variables.

Let's say you have two variables $$x1$$ and $$x2$$, and 6 samples. I could easily find two clusters with techinques such as k-Means.

When it comes to text mining, these variables are frequently associated with word frequency and/or context representation. For example, a sample may be a document and $$x1$$ and $$x2$$ may be the frequency of word "a" and word "b". Also, timestamp could be $$x3$$.

If you want to find clusters among documents, you need first to define a variable or "feature" extraction method (such as word frequency, tf-idf, word embedding etc). You can concatenate your text features with time-related features, and apply any clustering technique to this set of features in order to cluster your documents.

@Peter suggest you to use a topic modelling technique, which is a method for reducing the feature dimensional space (2 features = 2 dimensions, 1000 features = 1000 dimensions) after applying a word frequency feature extraction. It will help you describe each of these documents according to the frequency of a certain set of important words. Roughly speaking, a topic is a set of words that appear together. So for each document, the topics will have a relevance level.

It is not strictly a "clustering" approach, but it certainly achieves document clusterization by using the most relevant topic.

If you want to have temporal dimension coupled with topic modelling, you have to study a little more, read some papers and have some practice with these methods I mentioned. It's possible to have a pipeline with topic modelling for the word frequency ($$x1$$ is the relevance of topic 1 for a document, $$x2$$ for topic 2 ...) then you attach the timestamp to the result and apply a clustering technique such as k-Means.

• Please, see my question update. Apr 6, 2020 at 9:01
• You need to think that "temporal nature" is another set of variables. You can attach "temporal" variables to your feature set and perform the cluster normally. Think about what if you have other document information (such as the age of the writer), it is reasonable to say that you will need a different approach for clustering itself just because you have this new information? No, the clustering process is the same, you only have a few more variables that you need to extract from the timestamp. Apr 6, 2020 at 16:15