# Imbalanced data causing mis-classification on multiclass dataset

I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase).

Structure or format of my data is as follows.

----------------------------------------------------------------------------------------
| product_title          | Key_value_pairs                               | taxonomy_id |
----------------------------------------------------------------------------------------
Samsung S7 Edge        | Color:black,Display Size:5.5 inch,Internal    | 211
Storage:128 GB, RAM:4 GB,Primary Camera:12 MP

Case cover Honor 8     | Color:transparent,Height:15 mm,width:22 mm    | 212

Ruggers Men's T-Shirt  | Size:L,ideal for:men,fit:regular,             | 111
sleeve:half sleeve

Optimum Nutrition Gold | Flavor:chocolate,form:powder,size:34 gm       | 311
Standard Whey Protein


Data distribution is not normal; it is highly imbalanced:

-------------------------
| taxonomy_id |   count |
-------------------------
111 |  851750
112 |  355592
113 |  379433
114 |   23138
115 |  117735
116 |  145757
117 | 1339471
121 |  394026
122 |  193433
123 |   78299
124 |  111962
131 |    1776
132 |    4425
133 |     908
134 |   23062
141 |   22713
142 |   42073
211 |    7892
212 | 1574744
221 |    1047
222 |  397515
223 |   53009
231 |    1227
232 |    7683
251 |     739
252 |     327
253 |   38974
254 |      25
311 |    2901
321 |    7126
412 |     856
421 |  697802
422 |  414855
423 |   17750
425 |    1240
427 |     658
429 |    1058
431 |   20760
441 |     257


As you can see they are highly imbalanced and leading to mis-classifications.

Steps I have performed till now

1) Merge product_title and key_value_pairs column and remove stop words and special characters and perform stemming.

2) I have used pipeline for TFIDFvectorizer(), LinearSVC()

vectorizerPipe = Pipeline([
('tfidf', TfidfVectorizer(lowercase=True, stop_words='english')),
('classification', OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge'))),
])


After this I have fit pipeline and stored the classifier in pickle

prd = vectorizerPipe.fit(df.loc[:, 'description'], df.loc[:, 'taxonomy_id'])


On Testing side I have repeated step 1 as mentioned above and then load the pickle and use predict function

pd = cl.predict([testData])


Issues I am facing

1. A lot of products are being mis-classified into some other categories

Example: Ultimate Nutrition Prostar 100% Whey Protein should be classified into category 311 but my classifier is classifying it as 222 which is completely wrong.

2. I am not sure whether to use TFidfVectorizer() or Hashingvectorizer(), can you guys help me in selecting one of this along with their parameters?

3. Algorithm I am using is LinearSVC, is it a good choice for multi-class classification problems with large amount of data? Or should I use different algorithms?

4. As my data is highly imbalanced I tried random undersampling. The results were improved but they were still not up to the mark. Also I am not sure whether this is the right approach to perform random undersampling:

pipe = make_pipeline_imb(
HashingVectorizer(lowercase=True),
RandomUnderSampler(ratio={111: 405805, 112: 170431, 113: 241709, 114: 8341, 115: 50328, 116: 89445, 117: 650020, 121: 320803, 122: 162557, 123: 66156, 124: 36276, 131: 1196, 132: 3365, 133: 818, 134: 15001, 141: 6145, 142: 31783, 211: 24728, 212: 100000, 221: 791, 222: 8000, 223: 35406, 231: 785, 232: 3000, 251: 477, 252: 127, 253: 29563, 254: 33, 311: 2072, 321: 5370, 412: 652, 421: 520973, 422: 99171, 423: 16786, 425: 730, 427: 198, 429: 1249, 431: 13793, 441: 160},random_state=1),
OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge')))

5. I am new in machine learning so I have used this approach for text classification. If my approach is wrong then please correct me with right one.

(It would be great if you give suggestion or solution with examples as it will help me understand better).

***EDIT-1****

RndmFrst = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)
LogReg = LogisticRegression()
voting = VotingClassifier(estimators=[('LogReg ', LogReg), ('RndmFrst', RndmFrst)], voting='soft', n_jobs=-1)

pipe = Pipeline([('tfidf', TfidfVectorizer(ngram_range=(1,4), max_features=50000)), ('clf', voting)])

pipe = pipe.fit(df.loc[:,'description'], df.loc[:,'taxonomy_id'])
Preds = pipe.predict(test_data)

• I just saw that you tried under-sampling. Just fyi, Startified K-fold cross validation in Sci-Kit Learn also takes class distribution into account. – Kasra Manshaei Feb 16 '18 at 12:20

Nice question!

## Some Remarks

For imbalanced data you have different approaches. Most well-established one is resampling (Oversampling small classes /underssampling large classes). The other one is to make your classification hierarchical i.e. classify large classes against all others and then classify small classes in second step (classifiers are not supposed to be the same. Try model selection strategies to find the best).

I have got acceptable results without resampling the data! So try it but later improve it using resampling methods (statistically they are kind of A MUST).

TFIDF is good for such a problem. Classifiers should be selected through model selection but my experience shows that Logistic Regression and Random Forest work well on this specific problem (however it's just a practical experience).

You may follow the code bellow as it worked simply well then you may try modifying it to improve your results:

train = pd.read_csv(...)

# TFIDF Bag Of Words Model For Text Curpos. Up to 4-grams and 50k Features
vec = TfidfVectorizer(ngram_range=(1,4), max_features=50000)
TrainX = vec.fit_transform(train)
TestX = vec.transform(test)

# Initializing Base Estimators
clf1 = LogisticRegression()
clf2 = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)

# Soft Voting Classifier For Each Column
clf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)], voting='soft', n_jobs=-1)
clf = clf.fit(TrainX, TrainY)
preds = clf.predict_proba(TestX)[:,1]


Please note that the code is abstract so TianX, TrainY,TestX,etc should be properly defined by you.

## Hints

Be careful about what is StopWord. Practically many people (including myself!) made this mistake to remove stop words according to pre-defined lists. That is not right!

Stop words are corpus-sensitive so You need to remove stopwords according to information theoretic concepts (to keep it simple you need to know TFIDF kind of ignores your corpus-specific stopwords. If you need more explanation please let me know to update my answer).

VotingClassifier is a meta-learning strategy in the family of Ensemble Methods. They take benefit from different classifiers. Try them as they work pretty well in practice.

Voting schema simply takes the results of different classifiers and return the output of the one which has the highest probability to be right. So kind of democratic approach against dictatorship ;)

Hope it helps!

• Welcome! For intuitive resampling you may refer to the link i put for resampling. There is a step-by-step instruction. – Kasra Manshaei Feb 16 '18 at 12:50
• I am trying your solution, if i got stuck anywhere or in case of any doubts i will post in comment section. hope that will be fine for you! – outlier Feb 16 '18 at 12:57
• sure my friend ... good luck! – Kasra Manshaei Feb 16 '18 at 12:57
• if it worked then you may accept the answer :) – Kasra Manshaei Feb 16 '18 at 12:58
• @outlier since the answer has addressed your issue, kindly accept (and possibly upvote) it; answers take up valuable time for (volunteer) respondents – desertnaut Feb 17 '18 at 18:59