I am new to ML and trying to solve problem of text segmentation.

I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found a lot of info, but I don't understand how to properly run this task.

It looks like a classic problem and I cant find proper naming for it.

I am looking for help to find proper names for this problem, and, how to approach it with existing tools.

My initial thought was to use word embeddings -> sentence vectors with rolling average to detect changes in topics, but this approach does not work. What are other ways to solve this problem?


1 Answer 1


The problem you are describing is not a classic NLP problem.

There is a similar classic NLP problem called "topic modelling", which consists of discovering topics in a collection of text documents. Topics are defined by a list of words relevant to the topic itself. The most paradigmatic approach to this problem may be Latent Dirichlet Allocation (LDA). It is an unsupervised learning approach.

Your problem, nevertheless, has somewhat also been approached from a machine learning perspective, at least partially. I can refer you to the article Unsupervised Topic Segmentation of Meetings with BERT Embeddings by Meta. This is its abstract:

Topic segmentation of meetings is the task of dividing multi-person meeting transcripts into topic blocks. Supervised approaches to the problem have proven intractable due to the difficulties in collecting and accurately annotating large datasets. In this paper we show how previous unsupervised topic segmentation methods can be improved using pre-trained neural architectures. We introduce an unsupervised approach based on BERT embeddings that achieves a 15.5% reduction in error rate over existing unsupervised approaches applied to two popular datasets for meeting transcripts.

The authors released their source code at github.

To understand its contents, you will need to have some background on BERT, an NLP neural network based on the Transformer architecture's encoder part. On https://datascience.stackexchange.com/ you can find plenty of specific questions and answers about it (and you can ask more if you don't find your specific doubts).

  • 1
    $\begingroup$ Thanks for the suggestions! I will look into these approaches and hopefully understand more $\endgroup$ Commented May 30, 2023 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.