1
$\begingroup$

I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better?

Topic Modelling with Mallet

I noticed that mallet finds interesting topics, but these topics just match partially with the topic i want to identify. So the idea is to iterate over all interesting k and search for a topic that meets the requirements with the highest percentage.

  1. Find with elbow several (good coherence score) k topics (=list of good number of k)
  2. Have a look at terms with a high frequency and (manually) map these terms to desired topics (= word list for each searched topic)
  3. model topics for all good k and search for a good match with the word list

Example

I would like to classify/identify text that is about the topic "pollution of the environment"

  1. [17, 15, 20, 33, 41, 42, 55, 120] k => with good coherence score
  2. [dirt, pollution, emission, contamination] => high frequency, and these words would match with the topic
  3. found with k=33 a topic which match these keywords.

This could be used to classify a unseen text (and later on for labeling the text for supervised learning)

Topic "pollution of the environment"

[('pollution', 0.09756215849203013), ('dirt', 0.09028743250670891), ('emission', 0.05491609816030263), ('contamination', 0.02589802450774354), ('sea', 0.02088654660674448), ('ocean', 0.017281515729574187), ('climate', 0.014694946490348864), ('crisis', 0.013676484852403893), ('waste', 0.011380904652591419), ('smog', 0.009812797051311068)])

$\endgroup$
1
$\begingroup$

You can use LDA (Latent Dirichlet Allocation), as input data it only needs different collections of text. As output it generates for each document the main topics (Like: Document1:0.3% Topic1, 0.2% Topic5, 0.2% Topic8, 0.2% Topic12, 0.1% Topic3) and for each Topic the different words with the amounts they contribute to that Topic. You can then browse the words of the Topics and manually decide how to name the label of each Topic.

$\endgroup$
  • $\begingroup$ Thanks. GuidedLDA seems to be an approach to get more specific topics. I will try that out. $\endgroup$ – bartman99 Jul 2 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.