What is the difference between CountVectorizer() and Tokenizer() or are they the same?

Question

from sklearn.feature_extraction.text import CountVectorizer

from keras.preprocessing.text import Tokenizer

I am going through some NLP tutorials and realised that some tutorials use CountVectrizer and some use Tokenizer. From my understanding, I thought that they both use one-hot encoding but someone please clarify this.

What I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM, Naive Bayes.

Thank you

Answer 1

I thought that they both use one-hot encoding

These are utility to preprocess your text. Like any other utility, it has multiple options to tweak your text. You should explore all the parameters using the official docs.
I will explain one of these i.e. OHE vs Count

from sklearn.feature_extraction.text import CountVectorizer
corpus = [ 'This movie is bad.Too Bad', 'Awesome Movie. Too Awesome']
vectorizer = CountVectorizer(binary=True) #binary=False will make it Count
x = vectorizer.fit_transform(corpus)

import pandas as pd
df = pd.DataFrame(x.toarray(), columns=vectorizer.get_feature_names())
df

Our end goal is to create Features and each Feature has an indicator for its contextual meaning.

what I don't understand is why CountVectorizer is not used on Deep Learning models such as RNN and Tokenizer() is not used on ML Classifiers such as SVM,

When we are modeling a simple ML algorithm, we generally use scikit-learn.
So there is no point in adding Keras there.

Same is True for Deep learning.
Though, in this case, we have another reason too i.e. Deep Learning generally works on a large dataset. So, we mostly use the idea of embedding on our features. So it's better to use a Framework that provides an end-to-end solution