what is difference between set() and word_tokenize()?

Question

from nltk.tokenize import sent_tokenize ,word_tokenize

sentence = 'jainmiah I love you but you are not bothering about my request, 
            please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)

Actually both word_tokenize() and set(word_tokenize()) both returns same answers what makes the difference?

Answer 1

There are two differences between word_tokenize and set.

Word_tokenize

• Returns a list (try print(type(word_tok)))
• Returns all the tokens, regardless if there are duplicates

Set

• Returns a set (try print(type(set_all)))
• Returns all the unique tokens

Try this instead

sentence = 'jainmiah jainmiah jainmiah I love you but you are not bothering about my request, please yaar consider me for the sake'

word_tok = word_tokenize(sentence)
print(word_tok)

set_all = set(word_tokenize(sentence))
print(set_all)