I have text tokenized on the word level and few lists of phrases stored as tuples.

What would be the most efficient way to resegment (and store) the text into phrases?

For example, a sentence like:

"He didn't work hard."

Which is now tokenized as:

"He" "did" "n't" "work" "hard" "."

Would become:

"He" "did" "n't" "work hard" "."

Assuming that "work hard" would be one of the phrases I have.

  • $\begingroup$ Welcome to DataScienceSE. Can you please give an example? $\endgroup$
    – Erwan
    Jul 19, 2021 at 21:21
  • $\begingroup$ Sure, the text file is tokenized into words, and I have a list of tuples with each of which contains phrases. What I would like to get is the same text but tokenized into phrases whenever that would be possible, so with a sentence: "The young couple was looking into the moon." Let's say that "the young couple" and "into the moon" are in my phrase-set, the resegmented text would be something like this (instead of: token token token; where token is mostly words): "the young couple" "was" "looking" "into the moon" (4 tokens instead of 8 in this case). $\endgroup$
    – Karvin
    Jul 19, 2021 at 22:25
  • $\begingroup$ Actually, seems like MWETokenizer may be what I'm looking for... $\endgroup$
    – Karvin
    Jul 19, 2021 at 23:01
  • $\begingroup$ I think that the most direct way would be to (1) tokenize the phrases, (2) search for the tokenized phrases in the tokenized text. $\endgroup$
    – Erwan
    Jul 20, 2021 at 8:40
  • $\begingroup$ Thank you Erwan, I already have them tokenized. I guess what I was asking was how would you go through with marking these phrases (combining into 1 token). $\endgroup$
    – Karvin
    Jul 22, 2021 at 0:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.