0
$\begingroup$
from sklearn.feature_extraction.text import CountVectorizer

from keras.preprocessing.text import Tokenizer

I am going through some NLP tutorials and realised that some tutorials use CountVectrizer and some use Tokenizer. From my understanding, I thought that they both use one-hot encoding but someone please clarify this.

What I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM, Naive Bayes.

Thank you

$\endgroup$
3
  • 1
    $\begingroup$ Does this stackoverflow post answer your question? $\endgroup$ – noe Jan 29 at 20:58
  • $\begingroup$ kind of but I am still not sure if they both do the same thing. Do they both use the one-hot encoding method? $\endgroup$ – tamilini Jan 30 at 17:38
  • $\begingroup$ what i don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM, Naive Bayes. $\endgroup$ – tamilini Feb 2 at 15:56
1
$\begingroup$

I thought that they both use one-hot encoding

These are utility to preprocess your text. Like any other utility, it has multiple options to tweak your text. You should explore all the parameters using the official docs.
I will explain one of these i.e. OHE vs Count

from sklearn.feature_extraction.text import CountVectorizer
corpus = [ 'This movie is bad.Too Bad', 'Awesome Movie. Too Awesome']
vectorizer = CountVectorizer(binary=True) #binary=False will make it Count
x = vectorizer.fit_transform(corpus)

import pandas as pd
df = pd.DataFrame(x.toarray(), columns=vectorizer.get_feature_names())
df

enter image description here

Our end goal is to create Features and each Feature has an indicator for its contextual meaning.

what I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM,

When we are modeling a simple ML algorithm, we generally use scikit-learn.
So there is no point in adding Keras there.

Same is True for Deep learning.
Though, in this case, we have another reason too i.e. Deep Learning generally works on a large dataset. So, we mostly use the idea of embedding on our features. So it's better to use a Framework that provides an end-to-end solution

$\endgroup$
1
  • $\begingroup$ thank you for your help. It is very informative. $\endgroup$ – tamilini Feb 3 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.