# How do I get ngrams for all combinations of words in a sentence?

Lets say I have a sentence "I need multiple ngrams". If I create bigrams using Tf idf vectorizer it will create bigrams only using consecutive words. i.e. I will get "I need", "need multiple", "multiple ngrams".

How can I get "I mutiple", "I ngrams", "need ngrams"?

• ngrams by definition are continuous. "I need", "need multiple", "multiple ngrams" aren'r ngrams. You could write a simple logic for this yourself. Commented Aug 25, 2021 at 7:23
• I posted an answer to get your desired output. But I agree with Palak. These aren't real ngrams, and it begs the question, why do you really need this? Commented Aug 25, 2021 at 7:25
• I am creating a training dataset for a binary classifier where I am trying to create features using combination of words. Tf idf vectorizer creates vectors for ngrams using conecutive words. I was thinking if it is possible to create vectors for combination of non consecutive words using tf idf vectorizer? Commented Aug 25, 2021 at 7:49

You can use itertools.combinations().

For example:

s = "I need multiple ngrams"
s_list = s.split(" ") # Assumes you tokenize with white space.

import itertools

combinations = list(itertools.combinations(s_list, 2)) # the second argument ("2" in this case) is the size of the n-gram.


You will get the following output:

[('I', 'need'), ('I', 'multiple'), ('I', 'ngrams'), ('need', 'multiple'), ('need', 'ngrams'), ('multiple', 'ngrams')]


You can use this code as well:

s = "I need multiple ngrams"
tokens = s.split(' ')

res = [(tokens[i],tokens[j]) for i in range(len(tokens) -1) for j in range(i+1, len(tokens))]


Output:

[('I', 'need'), ('I', 'multiple'), ('I', 'ngrams'), ('need', 'multiple'), ('need', 'ngrams'), ('multiple', 'ngrams')]