# Accuracy reduces drastically when using TruncatedSVD with hashingvector

I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given product description. Since the dataset was large I was not able to able TF-IDF on that data it throws MemoryError.

I found that Hashingvector was desirable when dealing with large data. But when Hashingvector was applied I found it produced data with 1048576 features. It took around 1 hour to train and SGD model and produced 78% accuracy.

Code:

import pandas as pd
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import SGDClassifier
from sklearn.calibration import CalibratedClassifierCV

data_file = "Product_description.txt"
#Reading the input/ dataset
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

train_data, test_data, train_label,  test_label = train_test_split(data.Product_description, data.Category,
test_size=0.3, random_state=100, stratify=data.Category)

sgd_model = SGDClassifier(loss='hinge', n_iter=20, class_weight="balanced", n_jobs=-1,
random_state=42, alpha=1e-06, verbose=1)
vectorizer = HashingVectorizer(ngram_range=(1,3))
data_features = vectorizer.fit_transform(train_data.Product_description)
sgd_model.fit(data_features, train_label)
test_data_feature = vectorizer.transform(test_data.Product_Combined_Cleansed)
Output_predict = sgd_model.predict(test_data_feature)
print(accuracy_score(test_label, Output_predict))


Output:

Accuracy 77.01%


Since the dimension was high I thought reducing the dimension would increase the accuracy and reduce the training time. I used TrancatedSVD to reduce the dimension but this drastically reduced the prediction accuracy but reduced the training time to 10 minutes.

Code2:

from sklearn.decomposition import TruncatedSVD
clf = TruncatedSVD(100)
clf.fit(data_features)


Output:

Accuracy 14%


Edit:

When I tried TruncatedSVD with 1000 as limit it throws memory error so only I opted to use 100 as the limit.

It is stated that reducing n_features on HashingVector will cause collisions there can be collisions: distinct tokens can be mapped to the same feature index. However, in practice, this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems) in scikit site.

I got optimum accuracy when I used ngram between 1 to 3 so only used that.

• Well, if you use dimensionality reduction, then it also makes collisions, doesn't it? Think of smaller hash size as dimensionality reduction (you'd be able to use much bigger dimensions in TSVD though) – Jakub Bartczuk May 30 '18 at 13:31
• Okay I will try reducing the HashSize and will let you know. Thanks – The6thSense May 30 '18 at 13:32
• I got 73% accuracy when I kept the n_feature to 1,00,000 above it improved very slowly. The running time reduced to 20 minutes. – The6thSense May 31 '18 at 9:06

## 2 Answers

Some possible directions:

• when you're unsure how your BoW model works check different n-gram ranges (did you even check how it works with only unigrams?)
• TruncatedSVD by default only iterates 5 times (n_iter), so you could increase that
• It's not at all surprising to get such drastic change of classification quality for such low dimensionality. Some questions you could ask are why 100 dimensions? Does TSVD work 1000 dimensions or something like that? What is the reconstruction error (how does TSVD's explained_variance_ratio_ look like)?
• HashingVectorizer also has parameter that can control number of features, namely n_features.
• Please find my comment in the edit. I will try increasing the n_iter and let you know – The6thSense May 30 '18 at 13:28

Reducing from 1048576 to 100 features makes more than a 99.99% reduction of the input features dimension, knowing that SVD does not focuses on finding interesting features for classification, but creates an uncorrelated vector space from the initial one.

You should try to increase the dimension resulting from the SVD (have you looked at the retained variance for 100 components ?) but more specifically I suggest you to use a feature selection pipeline using a metric such as chi2 score, which gives better results for a classification goal (it may however be longer to compute).

• Please find my comments in edit. I will try Chisquare and let you know. – The6thSense May 30 '18 at 13:27