I have a .json file as dataset of the type:
and I am working on a classification problem in which I have to predict 4 classes, which are rhe semantic. I have worked through the problem, and after splitting the dtataset into training and test sets, I get an accuracy of $1$ .I have an unbalanced dataset, so I have oversampled it:
my code is the following:
dataFrame = pd.read_json('dataset.json',lines = True)
df = dataFrame[["lista_asm", "semantic"]].copy()
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf_vectorizer=TfidfVectorizer()
df_x = df['lista_asm']
X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = df['semantic']
#oversampling
from imblearn.over_sampling import RandomOverSampler
ros = RandomOverSampler()
X_ros, y_ros = ros.fit_sample(X_all, y_all)
print(X_ros.shape[0] - X_all.shape[0], 'new random picked points')
#splitting
X_train, X_test, y_train, y_test = train_test_split(X_ros, y_ros,
test_size=0.2, random_state=15)
from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
#fitting the model
clf = svm.SVC(kernel='linear', C=10).fit(X_train,y_train)
y_pred = clf.predict(X_test)
#checking accuracy
acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc) #from here I get accuracy 1
I have not written the imports in order to avoid making the code too long here, but if needed I can add them.
So I am not testing on the training set, but the results are too good so for sure there mmust be something wrong.I don't understand what is wrong.
Is this something that happen for any specific reason?
I have tried to change some stuff to make the code worse, in order to see if the accuracy would go down, but it remains to one.
[EDIT]I have tried to use your suggestions, and I am already sorry since for sure I am doing something wrong.
Here is what I have done:
X_all = df['lista_asm']
y_all = df['semantic']
X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
test_size=0.2, random_state=15)
tf = TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf.fit(X_train)
xtrain = tf.fit_transform(X_train)
from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
clf = svm.SVC(kernel='linear', C=10).fit(xtrain,y_train)
tf2=TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True,
max_features = 20000)
tf_transformer = tf2.fit(X_test)
X_test = tf2.fit_transform(X_test)
y_pred = clf.predict(X_test)
acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc) #gives accuracy 0.277
and if I plot the other performance metrics:
with the following message:
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use
zero_division
parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result))
[EDIT]I have also tried to use the same approch but with a .jsonl file, and in this case it works fine giving a reasonable accuracy. Maybe it is wrong to do so, but could the problem be here, so that I am not handling correctly a .json file?
Can somebody please help me?