I'm building ngrams after removing punctuation and lemmatization. The algorithm is to detect keywords in large bodies of text.

I have concerns that 2 documents

The child played with the red ball.


The sign was red. Balls are the toys of children.

would both contain "red ball". Are there best practices here? Ideally I would not want the second document to have the same value of "red ball" as the second.


The n-gram model is often built after segmenting into words and sentences. If the data is segmented by sentence it's easy to avoid any overlap between sentences: one can simply extract n-grams sentence by sentence independently. In case it's more convenient to extract all the n-grams at once, padding can be used to mark the beginning/end of sentences like this:

The sign was red #SENT# Balls are the toys of children #SENT#

Dealing with other punctuation signs which don't mark the end of a sentence can be a bit more tricky, especially if you want to keep the possibility of a keyword which spans over some punctuation signs (for instance in "red-handed" or "tl;dr").

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.