I'm having trouble understanding why I would use dropout, regularization, data augmentation, etc to get rid of overfitting in the first place. I get that if your model is too large or data is too sparse then your model may start to memorize data and not perform well on new data. However, are there any cases in which adding dropout, regularization, etc would increase accuracy on the validation set? For instance, if my training acc is 95% and val accuracy is 70%, would removing overfitting simply bring the training accuracy down lower to the val accuracy? Or is there a way to actually improve training accuracy? I assume there is but some intuition on this would be very much appreciated!
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Its like studying for an exam with only Past year papers (PYP) and you are the classifier. It would not be wise not to practice any PYP all for the exam, leading to poor performance in the exam (under-fitting). On the other hand, it would be terrible to memorize the answers to the PYP as you cannot generalize well to the exam paper which is definitely different than the PYP (over-fitting).
$\begingroup$
$\endgroup$
Basically, prediction error can be decomposed into three terms, two of which that you can control. MSE (mean square error) = Variance + Bias^2 + irreducible error. A model with high variance implies that small changes in the training dataset yield large changes in the fit of the model (in this context). A model with low bias essentially fits the training dataset extremely well. Irreducible error is the difficulty inherent to the problem itself (and therefore, not controllable) so lets ignore that.
In an ideal world, we would obviously want low variance AND low bias. However, this in reality is extremely difficult to do because intuitively, the two forces of variance and bias work against each other. The reason why is that I can arbitrarily make my model fit the data extremely well (and therefore, have low bias) by simply increasing the complexity of the model, modelling every quirk and essentially do what you refer to as "memorizing the dataset". You see, data are realizations of random variables that can potentially take on many different outcomes and therefore, there is noise in the dataset. Hence, modelling the exact pattern/trend in the dataset is not going to generalize well because the data itself is not a perfect representation of the population (which is our goal) but a sample of it.
By increasing the complexity of my model to arbitrarily improve the fit of the model on the dataset, I have effectively traded away all of my bias for large amounts of variance. I did this because clearly, by increasing the complexity of my model to the point where my model has even memorized the noise in the dataset, a relatively small change in the dataset with different observations (another random sample from the same population) will change the fitted model drastically. This is because noise is inherently random, not predictable and unique to that specific dataset (sample) only, not the general population from which the data comes from. As a consequence of my model changing a lot with my dataset that I trained it with (what many refer to as an unstable model), my models predictions will inherently be very random as well and therefore, my models performance on new held out data will be no better than say, a random guesser. This is what people mean when they say their model is "overfit". I've fit the training data set perfectly, at the cost of not modelling the actual signal from the data but a bunch of noise unique to the training set only.
So what people have figured out is that we need to make a tradeoff between fitting the training data extremely well whilst also being mindful of not modelling noise. Thus, stuff like dropout layers in NN, L1/L2 regularization, pruning in trees, learning rates, early stopping, and cost complexity penalties all work in purposely sacking some bias (fitting the training set not as well) in hopes of reducing variance (having a stable model and therefore, stable non random predictions). Striking the balance between the two is in essence what predictive modelling is about and all ML methods try to address this problem in their own way.
In your example, removing overfitting would imply moving the validation accuracy closer to the training accuracy, not the other way around. But of course, this is typically a pipe dream and you will almost always overfit somewhat. The degree to what you find acceptable, given the problem at hand, is key. If I have a training accuracy of 0.99 and a validation accuracy of 0.89 in many contexts I would still be satisfied with the model, for example.
$\begingroup$
$\endgroup$
Training accuracy is nearly irrelevant. To answer some of your questions, I will use a hypothetical example.
Let us pretend we are doing binary classification. For simplicity we have perfectly balanced classes (i.e. same number of class A instances as class B instances).
Model 1
We train a model and get 100% training accuracy, but 50% validation accuracy. This model is worthless! Even though it gets 100% training accuracy, it is as good as a coin flip when tested on unseen data.
Model 2
We train another model and get 80% training accuracy and 70% validation accuracy. This is a great improvement on our first model, even though our training accuracy drops 20%.
Model 3
Our third model gets 55% training accuracy and 60% validation accuracy. This might be due to a small validation set, which would make the validation accuracy rely more on chance. Even though our end goal is to maximize validation accuracy, our model still needs to generalize our feature space, which would inherently lead to greater training accuracies!
Now let us visualize (please excuse my crude drawings) model generalizations for binary classification. Each instance has 2 features, which together define our feature space. Our model tries to separate the two classes with a single continuous line in this feature space (green line is our model). Our training and validation data sets are plotted below.
Here we can see that increasing training accuracy does not directly lead to increasing validation accuracy (see over fitting example). However, training accuracy still plays a role in the validation accuracy if, for example, there is underfitting.
Remember our goal: maximize validation accuracy.
Both underfitting and overfitting led to worse performance on the validation set compared to the models with good fits. This is the reason we do not want overfitting (or underfitting); both tend to decrease validation accuracy.
In general, validation accuracy is less than or around training accuracy. It is rarely much greater than training accuracy, because that often implies there was significant luck. Therefore, although our goal is to maximize validation accuracy, the training accuracy acts almost like a ceiling for the validation accuracy. This is why I said that it is nearly irrelevant:
It is not our end goal to maximize training accuracy, but doing so is usually a prerequisite to achieving better validation accuracy.
Underfitting
Overfitting
Good Fits
