LinearSVC training time with CountVectorizer and HashingVectorizer

Question

I am currently trying to build a text classifier and I am experimenting with different settings. Specifically, I am extracting my features with a CountVectorizer and HashingVectorizer:

from sklearn.feature_extraction.text import CountVectorizer, HashingVectorizer

# Using the count vectorizer.
count_vectorizer = CountVectorizer(lowercase=False, ngram_range=(1, 2)) 
X_train_count_vectorizer = count_vectorizer.fit_transform(X_train['text_combined'])
X_dev_count_vectorizer = count_vectorizer.transform(X_dev['text_combined'])

# Using the has vectorizer.
hash_vectorizer = HashingVectorizer(n_features=2**16,lowercase=True, ngram_range=(1, 2))
X_train_hash_vectorizer = hash_vectorizer.fit_transform(X_train['text_combined'])
X_dev_hash_vectorizer = hash_vectorizer.transform(X_dev['text_combined'])

Then I am using a LinearSVC classifier

from sklearn.svm import LinearSVC

# Testing with CountVectorizer.
clf_count = LinearSVC(random_state=0)
clf_count.fit(X_train_count_vectorizer, y_train)
y_pred = clf_count.predict(X_dev_count_vectorizer)
accuracy_score(y_dev, y_pred)

# Testing with HasingVectorizer.
clf_count = LinearSVC(random_state=0)
clf_count.fit(X_train_hash_vectorizer, y_train)
y_pred = clf_count.predict(X_dev_hash_vectorizer)
accuracy_score(y_dev, y_pred)

I obtained the following results:

	Time to train	Accuracy
CountVectorizer	59.9 seconds	83.97%
HashingVectorizer	6.21 seconds	84.92%

Please note that even when limiting the number of features of the CountVectorizer to 2**18, I still get slower training and inferior reults.

My questions:

• Why is training with CountVectorizer slower even for a similar number of features?
• What could explain the performance gain in terms of training time?
• Any intuition on the reasons behind the accuracy gain?

For my particular case, I have also trained a TfidfVectorizer and the CountVectorizer worked a bit better. If the HashingVectorizer has such significant advantages in certain cases. I am wondering why the HashingVectorizer usage is not more widely introduced in different NLP tutorials?

Answer 1

Your lowercase setting is different for CountVectorizer and HashingVectorizer. It might have an impact.

Otherwise, they do very similar job in this case, the accuracy difference varies with the exact task but is not that huge. Disparate training speeds you observe are not related to the method itself (the feature matrix size is comparable), it's just HashingVectorizer normalizing the results by default, which is usually beneficial for SVC, resulting in much fewer iterations (check clf_count.n_iter_). Applying sklearn.preprocessing.Normalizer() to CountVectorizer results will likely make it fit equally fast.

HashingVectorizer is still faster and more memory efficient when doing the initial transform, which is nice for huge datasets. The main limitation is its transform not being invertible, which limits the interpretability of your model drastically (and even straight up unfitting for many other NLP tasks).