I have a .json file as dataset of the type:
and I am working on a classification problem in which I have to predict 4 classes, which are rhe semantic. I have worked through the problem, and after splitting the dtataset into training and test sets, I get an accuracy of $1$ .I have an unbalanced dataset, so I have oversampled it:
my code is the following:
dataFrame = pd.read_json('dataset.json',lines = True) df = dataFrame[["lista_asm", "semantic"]].copy() from sklearn.feature_extraction.text import TfidfVectorizer tfidf_vectorizer=TfidfVectorizer() df_x = df['lista_asm'] X_all = tfidf_vectorizer.fit_transform(df_x) y_all = df['semantic'] #oversampling from imblearn.over_sampling import RandomOverSampler ros = RandomOverSampler() X_ros, y_ros = ros.fit_sample(X_all, y_all) print(X_ros.shape - X_all.shape, 'new random picked points') #splitting X_train, X_test, y_train, y_test = train_test_split(X_ros, y_ros, test_size=0.2, random_state=15) from sklearn import svm from sklearn.model_selection import cross_val_score from sklearn.ensemble import RandomForestClassifier #fitting the model clf = svm.SVC(kernel='linear', C=10).fit(X_train,y_train) y_pred = clf.predict(X_test) #checking accuracy acc = clf.score(X_test, y_test) print("Accuracy %.3f" %acc) #from here I get accuracy 1
I have not written the imports in order to avoid making the code too long here, but if needed I can add them.
So I am not testing on the training set, but the results are too good so for sure there mmust be something wrong.I don't understand what is wrong.
Is this something that happen for any specific reason?
I have tried to change some stuff to make the code worse, in order to see if the accuracy would go down, but it remains to one.
[EDIT]I have tried to use your suggestions, and I am already sorry since for sure I am doing something wrong.
Here is what I have done:
X_all = df['lista_asm'] y_all = df['semantic'] X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, test_size=0.2, random_state=15) tf = TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True, max_features = 20000) tf_transformer = tf.fit(X_train) xtrain = tf.fit_transform(X_train) from sklearn import svm from sklearn.model_selection import cross_val_score from sklearn.ensemble import RandomForestClassifier clf = svm.SVC(kernel='linear', C=10).fit(xtrain,y_train) tf2=TfidfVectorizer(analyzer='word', ngram_range=(1,2), lowercase = True, max_features = 20000) tf_transformer = tf2.fit(X_test) X_test = tf2.fit_transform(X_test) y_pred = clf.predict(X_test) acc = clf.score(X_test, y_test) print("Accuracy %.3f" %acc) #gives accuracy 0.277
and if I plot the other performance metrics:
with the following message:
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use
zero_divisionparameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result))
[EDIT]I have also tried to use the same approch but with a .jsonl file, and in this case it works fine giving a reasonable accuracy. Maybe it is wrong to do so, but could the problem be here, so that I am not handling correctly a .json file?
Can somebody please help me?