from nltk.tokenize import sent_tokenize ,word_tokenize
sentence = 'jainmiah I love you but you are not bothering about my request,
please yaar consider me for the sake'
word_tok = word_tokenize(sentence)
print(word_tok)
set_all = set(word_tokenize(sentence))
print(set_all)
Actually both word_tokenize() and set(word_tokenize()) both returns same answers what makes the difference?