0
$\begingroup$

As the title say my code produces low train accuracy and high test accuracy.

First I split my data set to train and test sets.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

I then fitted my model with the training sets

model = LogisticRegression(max_iter=10000, penalty='l2')
model.fit(X_train, y_train)

After which I printed the Training accuracy of the model

predictions = model.predict(X_train)
print('Training Accuracy: {}'.format(accuracy_score(y_train, predictions)))

The training accuracy showed 0.8275. I then validated my model using stratified KFold cross validation and it gave me an 0.8125 mean accuracy score.

cross_val = StratifiedKFold(n_splits=10, random_state=1, shuffle=True)
scores = cross_val_score(model, X_train, y_train, scoring='accuracy', cv=cross_val, n_jobs=-1)

for index, score in enumerate(scores):
    print('Iteration {} Accuracy score: {}'.format(index + 1, score))
    
print('\nMean Accuracy: {}'.format(np.mean(scores)))

After that I evaluated by testing set with the following code:

testing_predictions = model.predict(X_test)

print(classification_report(y_test,testing_predictions))

cm = confusion_matrix(y_test, testing_predictions, labels=model.classes_)
display = ConfusionMatrixDisplay(confusion_matrix = cm, display_labels=model.classes_)
display.plot()

enter image description here

Surprisingly it gave me a higher testing accuracy of 0.86 than the training accuracy, 0.82. I have also red that it is impossible to have high testing accuracy than the training accuracy. Did I do something wrong in my process?

$\endgroup$

2 Answers 2

3
$\begingroup$

With your sample size I can see that happening by chance... rerun your train/test split 100 times (drop random seed) and see how often the performance is better on the test set.

$\endgroup$
3
  • $\begingroup$ Oh I see! now my mean train score is 0.8361 and my mean test score is 0.8246 It does not overfit. $\endgroup$ Feb 5 at 7:04
  • $\begingroup$ Incidentally, how do you know if your sample size is to small? $\endgroup$
    – eliangius
    Feb 5 at 14:02
  • $\begingroup$ It's not that it's too small ... it's that there is always uncertainty around measuring performance - and that the larger your sample the less uncertainty there is. Check out bootstrapped sampling and calculating confidence intervals for more information. $\endgroup$ Feb 5 at 17:08
0
$\begingroup$

It is definitely possible to have a testing accuracy that is higher than the training accuracy. If the difference between the two is relatively large you are probably underfitting your model, i.e. not using all of the signal that is in the data. In your case the difference is quite small, so I don't see anything wrong with your approach. It just means that you are likely successful in preventing your model from overfitting on your data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.