- First point: in general it's risky to use accuracy in order to measure performance, especially if there is class imbalance. F1-score would be a better option in general.
- Significantly lower performance on the test set compared to the training set is indeed the main indication of overfitting. It's worth keeping in mind that Overfitting means that the model captures too many details which happen by chance in the training set, a problem more likely to happen if the training set is too small and/or the model is too complex.
but why would I even check the accuracy for the training dataset?
Simply to detect overfitting, because it's a very common issue.
If I have a good accuracy on the testing dataset that means I'm most likely not overfitting, right? (Assuming that we make sure that the model doesn't train on any testing data)
This is correct, but very often we don't know how to define "good": for a very hard task, a F1-score of 0.5 could be very good for instance, but for some very easy task an accuracy of 0.99 might be bad. In other words, unless there is some benchmark performance to compare against, there's no easy way to know if the performance is good.
I have another question: Can oversampling using SMOTE cause overfitting (good accuracy on the testing dataset but in reality it is overfitting?) SMOTE draws a line and makes new points on it so it doesn't duplicate the data.
Yes, totally: the points artificially created by SMOTE are not real data points, so there is a high chance that they do not have the same variations that real data points would have (whether it's noise or significant patterns). But the number of points can mislead the model, which is likely to assume that there's an important regular pattern, even though it's actually only caused by very few (possibly random) points artificially inflated by SMOTE.
