Thematic clustering of text

Question

Please advise on starting points, research (papers,frameworks) related to thematic clustering of text. In particular on a system with two levels of clustering where second level has a temporal nature. Thanks!

Update:

Sorry for ambiguity in my initial question. I need to clarify, that I have experience with clustering in general and document clustering in particular, for text using TFIDF, word embeddings (word2vec, Glove and BERT Sentence embeddings) in a vector space.

My question originates from some text mentioning "two levels of thematic clustering of text where second level has a temporal nature". Just wanted to know about this technique and in particular "temporal nature" used in clustering.

Answer 1

Your question is way to vague. What did you try, what do you want to achieve in detail. However, find below some R examples how to approach topic modeling. This book might also be helpful: Text Mining with R.

library(topicmodels)
data("AssociatedPress")
AssociatedPress

# LDA (Latent Dirichlet Allocation)
ap_lda <- LDA(AssociatedPress, k = 2, control = list(seed = 1234))
ap_lda

# Get "topics"
library(tidytext)
ap_topics <- tidy(ap_lda, matrix = "beta")
ap_topics

library(ggplot2)
library(dplyr)

# Plot topics
ap_top_terms <- ap_topics %>%
  group_by(topic) %>%
  top_n(10, beta) %>%
  ungroup() %>%
  arrange(topic, -beta)

ap_top_terms %>%
  mutate(term = reorder(term, beta)) %>%
  ggplot(aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free") +
  coord_flip()

# (words with) Greatest difference between categories
library(tidyr)

beta_spread <- ap_topics %>%
  mutate(topic = paste0("topic", topic)) %>%
  spread(topic, beta) %>%
  filter(topic1 > .001 | topic2 > .001) %>%
  mutate(log_ratio = log2(topic2 / topic1))

beta_spread

list(beta_spread$term)

# Select 10 largest and smallest values of "log_ratio")
df1 <- subset(beta_spread, beta_spread$log_ratio<=max(tail(sort(beta_spread$log_ratio, decreasing = T),10)))
df2 <- subset(beta_spread, beta_spread$log_ratio>=min(tail(sort(beta_spread$log_ratio, decreasing = F),10)))
df = rbind(df1,df2)
df

# Plot
barplot(sort(df$log_ratio), names.arg=df$term[order(df$log_ratio)], las=2)

# Gamma contains the probability of document X to belong to topic Y
ap_documents <- tidy(ap_lda, matrix = "gamma")
ap_documents
# the model estimates that only about 24.8% of the words in document 1 were generated from topic 1

Answer 2

"Clustering" is a very broad umbrella for a set of unsupervised techniques that tries to group data items together, according to its characteristics/covariates/variables.

Let's say you have two variables $x1$ and $x2$, and 6 samples. I could easily find two clusters with techinques such as k-Means.

When it comes to text mining, these variables are frequently associated with word frequency and/or context representation. For example, a sample may be a document and $x1$ and $x2$ may be the frequency of word "a" and word "b". Also, timestamp could be $x3$.

If you want to find clusters among documents, you need first to define a variable or "feature" extraction method (such as word frequency, tf-idf, word embedding etc). You can concatenate your text features with time-related features, and apply any clustering technique to this set of features in order to cluster your documents.

@Peter suggest you to use a topic modelling technique, which is a method for reducing the feature dimensional space (2 features = 2 dimensions, 1000 features = 1000 dimensions) after applying a word frequency feature extraction. It will help you describe each of these documents according to the frequency of a certain set of important words. Roughly speaking, a topic is a set of words that appear together. So for each document, the topics will have a relevance level.

It is not strictly a "clustering" approach, but it certainly achieves document clusterization by using the most relevant topic.

If you want to have temporal dimension coupled with topic modelling, you have to study a little more, read some papers and have some practice with these methods I mentioned. It's possible to have a pipeline with topic modelling for the word frequency ($x1$ is the relevance of topic 1 for a document, $x2$ for topic 2 ...) then you attach the timestamp to the result and apply a clustering technique such as k-Means.