1
$\begingroup$
from nltk.tokenize import sent_tokenize ,word_tokenize

sentence = 'jainmiah I love you but you are not bothering about my request, 
            please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)

Actually both word_tokenize() and set(word_tokenize()) both returns same answers what makes the difference?

$\endgroup$
1
$\begingroup$

There are two differences between word_tokenize and set.

Word_tokenize

  • Returns a list (try print(type(word_tok)))
  • Returns all the tokens, regardless if there are duplicates

Set

  • Returns a set (try print(type(set_all)))
  • Returns all the unique tokens

Try this instead

sentence = 'jainmiah jainmiah jainmiah I love you but you are not bothering about my request, please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.