I have a classification that has to predict three different classes: gcc,icc, clang. The prblem is that if I use a blind test set to do a submission, when I look athe the prediction I have on it I find that most of the predictions are only of one type. So I have the following situation:

enter image description here

My code is the following:

import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

dataFrame = pd.read_json('training_dataset.jsonl',lines = True)

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()[0] for i in x])

df = dataFrame[["opcodes", "compiler"]].copy()

df['opcodes'] = [" ".join(opcode) for opcode in df['opcodes'].values]

from sklearn.feature_extraction.text import TfidfVectorizer 


df_x = df['opcodes']

X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = dataFrame['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, 
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100).fit(X_train,y_train)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)    
print("Accuracy %.3f" %acc)

Now, using a random forest classifier I get an accuracy of 0.89, but the problem is that the model predicts mostly the class gcc, so I guess something is wrong.

I was almost soing a submission when I noticed this.

Does this happen because I didn't consider the fact that I have a multiclass classification problem? I really don't understand.

Can somebody please help me? Thank's in advance.

[EDIT] The dataset is balanced.

[EDIT 2] When try to do the prediction on the blind test set I do the following:

test = pd.read_json('test_dataset_blind.jsonl',lines = True)

test['instructions'] = [" ".join(opcode) for opcode in 

df_x_new = test['instructions']

X_new = tfidf_vectorizer.fit_transform(df_x)

new_pred_class = clf.predict(X_new)[:3000]

sub_compiler = pd.DataFrame({'instructions': 

[EDIT 4]anctually the problem seems to be on how I select the data to test. Infact, if I do new_pred_class = clf.predict(X_new)[-3000:], I get:

enter image description here ` so this time mostly the clang are predicted.

I have 30000 rows in the original dataset and 3000 in the blind test set.

[EDIT 5]So I think I have to reorganize the select the new_pred_class taking random values from clf.predict(X_new)[:3000], so what I mean is that I should take 3000 random values. How could I do that? Thank's again.

EDIT 6] if I do type(X_new) I have scipy.sparse.csr.csr_matrix

  • $\begingroup$ try clf.predict(X_new.sample(3000)) $\endgroup$ – oW_ Nov 9 '19 at 21:45
  • $\begingroup$ Thank's for the answer. It gives me the error : AttributeError: sample not found $\endgroup$ – J.D. Nov 9 '19 at 21:51
  • $\begingroup$ I solved using .resample(). Thank you. $\endgroup$ – J.D. Nov 9 '19 at 22:31

It is a common problem that - with unbalanced classes - some model tends to predict mostly the majority class. You could try to oversample the minority classes. In addition, RF tends to perform weak here. Boosting or NN are often able to recover more details, which can be important to predict the minority classes.

Edit: Okay, now that you clarified that you face a balanced problem, I guess your problem is the classifier. You could try multinominal logit with lasso regulation to „shrink“ irrelevant features (words). You should also consider Boosting or NN (maybe LSTM sequential).

  • $\begingroup$ Than's for your answer. In my case the dataset is balanced, so I have 10000 gcc,10000 icc and 10000 clang. $\endgroup$ – J.D. Nov 9 '19 at 21:11
  • $\begingroup$ do you use a test set made out of the available data? $\endgroup$ – Peter Nov 9 '19 at 21:29
  • $\begingroup$ the test set is a separate set from the beginning dataset. It contains only instances, not the classes $\endgroup$ – J.D. Nov 9 '19 at 22:01
  • $\begingroup$ ? ...but you can create your own test set from the data you have, right. And you should do that to validate your model. $\endgroup$ – Peter Nov 9 '19 at 22:23
  • $\begingroup$ I have already done the split and already used the model. This is a next step where I have a new test set and I have to test what I have done on this new set. Anyway I have solved the problem using .resample() and so I got randmo samples for testing. Thank you anyway for the answers. $\endgroup$ – J.D. Nov 9 '19 at 22:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.