I have a bunch of .txt and .srt files extracted from a MOOC website, they are the scripts of the videos. I would like to segment the scripts into parts such that each part falls into one of the following categories:
MainConceptDescription-> Explanation of the Main concept(s)
SubConceptDescription-> Explanation of Subconcept related to the main concept
Methodology / Technique-> To achieve something, what should one do
Summary-> Summary of the discussed material or of the whole course
Application-> Practical advise for the concept
Example-> Concept example

Now, for the first 2 i think I should try to apply Latent Dirichlet Allocation to extract the topics. Another idea was to look in the resource name and search for these words in the text. Another idea was to read some of the resources and manually fix some sort of a dictionary for every category and then create regex patterns and search for them in the text.

But the latter seems too lame. So now I am not sure what can I do. I've seen similar works for research papers, however research papers have their own specific expressions etc that are more or less constant and seen in most of the papers, but that is not the case with my video scripts where it's 100% spoken natural language I need to work on. Do you have any ideas how can I approach this? I do have a list of keywords that kind of signifies whether an example follows or a concept is explained, but I am doing this manually which is surely not what I want to do for 563 files that may as well become many more.

Furthermore, I'd like to connect topics found to ontologies in order to enrich the metadata about each file. I have no idea how can I approach this either. Any advice would be VERY appreciated.

Forgive me if my explanations make no sense. I am not too familiar with the terminology. So if you also explain some of the terminology you use, i'd appreciate that too. And please advise on algorithms I can try with.


1 Answer 1


I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your broader categories; description, methodology, classifier, application, and example. That is, I would manually label some transcripts at the paragraph level. If you don't have paragraph-segmented text, smooth the classification probability over the sentences so they share the same label. Or use a paragraph segmentation model as in Automatic Paragraph Segmentation with Lexical and Prosodic Features.

The classifier could be a CRF or an RNN. The modern way to induce the ontology would be through word embeddings; cf. e.g, Learning Semantic Hierarchies via Word Embeddings. Formerly I would have recommended a hierarchical topic model, such as hLDA; cf. e.g, Non-Parametric Estimation of Topic Hierarchies from Texts with Hierarchical Dirichlet Processes or Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling

Welcome to the site and good luck!

  • $\begingroup$ Thank you very much, @Emre! Do you know any good sources showing how to connect ontology with natural text? Or in general, as I am not very familiar with this. I consider trying MS Academic Graph, but I am having trouble using their services. I need something in the domain of Computer Science, Mathematics, Physics etc? WordNet I think is a bit too broad and may not work for more specific topics. $\endgroup$
    – A.D.
    Commented Apr 16, 2018 at 18:46
  • $\begingroup$ and regarding the paragraphs, I do not have paragraphs, I managed to split sentences each on a new line, but it was all one big text. Do you mean I should label every sentence to build a train set? $\endgroup$
    – A.D.
    Commented Apr 16, 2018 at 18:47
  • $\begingroup$ I edited my answer to show how to do paragraph segmentation in the first paragraph, and ontology induction in the second paragraph. $\endgroup$
    – Emre
    Commented Apr 16, 2018 at 18:56
  • $\begingroup$ Thank you very much! May I tag you with some questions, if I have, and if you don't mind? :) $\endgroup$
    – A.D.
    Commented Apr 16, 2018 at 19:07
  • 3
    $\begingroup$ Ask a new question; don't add comments here. $\endgroup$
    – Emre
    Commented Apr 16, 2018 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.