1
$\begingroup$

Lets say I have a sentence "I need multiple ngrams". If I create bigrams using Tf idf vectorizer it will create bigrams only using consecutive words. i.e. I will get "I need", "need multiple", "multiple ngrams".

How can I get "I mutiple", "I ngrams", "need ngrams"?

$\endgroup$
3
  • 3
    $\begingroup$ ngrams by definition are continuous. "I need", "need multiple", "multiple ngrams" aren'r ngrams. You could write a simple logic for this yourself. $\endgroup$ Commented Aug 25, 2021 at 7:23
  • $\begingroup$ I posted an answer to get your desired output. But I agree with Palak. These aren't real ngrams, and it begs the question, why do you really need this? $\endgroup$ Commented Aug 25, 2021 at 7:25
  • $\begingroup$ I am creating a training dataset for a binary classifier where I am trying to create features using combination of words. Tf idf vectorizer creates vectors for ngrams using conecutive words. I was thinking if it is possible to create vectors for combination of non consecutive words using tf idf vectorizer? $\endgroup$ Commented Aug 25, 2021 at 7:49

2 Answers 2

3
$\begingroup$

You can use itertools.combinations().

For example:

s = "I need multiple ngrams"
s_list = s.split(" ") # Assumes you tokenize with white space.

import itertools

combinations = list(itertools.combinations(s_list, 2)) # the second argument ("2" in this case) is the size of the n-gram.

You will get the following output:

[('I', 'need'), ('I', 'multiple'), ('I', 'ngrams'), ('need', 'multiple'), ('need', 'ngrams'), ('multiple', 'ngrams')]
$\endgroup$
1
$\begingroup$

You can use this code as well:

s = "I need multiple ngrams"
tokens = s.split(' ')

res = [(tokens[i],tokens[j]) for i in range(len(tokens) -1) for j in range(i+1, len(tokens))]

Output:

[('I', 'need'), ('I', 'multiple'), ('I', 'ngrams'), ('need', 'multiple'), ('need', 'ngrams'), ('multiple', 'ngrams')]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.