0
$\begingroup$

I'm new to data mining. I want to detect topic transition among consecutive sentences. For instance, I have a paragraph (this could be a collection of dozens of sentences, sometimes without transitional words) as follows:

As I really like Mickey Mouse, I was hopping to go to Florida. But my dad took me to Nevada. Obviously, Mickey Mouse was not there. But, I attended a camp with other children. And, I really enjoyed and learnt a lot from my camp.

Here, I want to automatically split this into following sub-paraphs:

  1. As I really like Mickey Mouse, I was hopping to go to Florida. But my dad took me to Nevada. Obviously, Mickey Mouse was not there.

  2. But, I attended a camp with other children. And, I really enjoyed and learnt a lot from my camp.

As far as I know, this is not the sentence similarity measurement. What technique should be used here? Any example using python or tensorflow models would be greatly appreciated.

$\endgroup$
6
  • $\begingroup$ Why not split before the first "but" in your exemple ? $\endgroup$ – lcrmorin Apr 28 '20 at 12:56
  • $\begingroup$ This is just an example, but there are some other paragraphs that does not contain transitional words. Besides, there're two 'but's in the example, and splitting based on it could lead to wrong answer. $\endgroup$ – E_learner Apr 28 '20 at 12:57
  • $\begingroup$ It is not clear from your exemple what would be the rule to make a split. Maybe try all possible cuts and choose the one that minimise a similarity measure between the two remaining paragraphs. $\endgroup$ – lcrmorin Apr 28 '20 at 13:00
  • $\begingroup$ Seems like a good solution, thanks. How to do similarity measurements between two paragraphs? I know there are some method for sentences, though. And, how to obtain all possible cuts? $\endgroup$ – E_learner Apr 28 '20 at 13:03
  • $\begingroup$ May be start with periods as possible cuts. Measuring similarity between two paragraphs is the difficult part. Maybe start with something like : text2vec.org/similarity.html#api $\endgroup$ – lcrmorin Apr 28 '20 at 13:29
2
$\begingroup$

One solution could be to:

  1. Get sentence embeddings from FastText
  2. Compute Euclidean Distance between the consecutive sentences
  3. If the distance between the consecutive sentences is close to 1, then, you may say the two sentences are talking about different topics.

See here how to compute sentence embeddings for the English language: https://github.com/facebookresearch/fastText/blob/5b5943c118b0ec5fb9cd8d20587de2b2d3966dfe/python/fasttext_module/fasttext/FastText.py#L127

fasttext.util.download_model('en', if_exists='ignore')  # English
ft = fasttext.load_model('cc.en.300.bin')
fasttext.util.reduce_model(ft, 20)
def get_fasttext_sentence_embedding(sentence, ft):
    if pd.isna(sentence):
        return np.zeros(20)
    return ft.get_sentence_vector(sentence)

Then, compute the euclidian distance between fast text embeddings of consecutive sentences.

The same can be done using LDA (topic model), but, that would require a lot of text to model the topics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.