# Is it possible to have a validation error less than train error for a while followed by the reverse behaviour?

I am solving for a regression (using tensorflow's DNNRegressor) problem. When I sampled out 20% data (randomly) and divided it further into train-eval (90-10%, random but mutually exclusive), I observed the following graphs-  Here, blue line corresponds to eval set loss and orange line is for train loss. The total length of data is nearly 4.8 million rows, thus 20% being nearly 1 million data points. How to interpret this behavior?

The behavior you notice in the graphs will always be the case regardless of the machine learning algorithm. Machine learning will require you to fit some parameters such that the model can capture the mappings from your input space to the desired outputs.

At the start of the training these parameters will all be assigned randomly, thus we should assume random performance, thus 50% for a 0/1 classification. As the model parameters are tuned using the training data, the model will tend to decrease both the training and evaluation loss. However, after some training there will be a point where the evaluation loss starts increasing. This is the point where the parameters are no longer generalizing for the data but is instead overfitting and specifically identifying points in the training data.

This is NOT what we want. This is overfitting. We want to stop the training at the minimum loss of the evaluation set.

Another way to think of this is if I ask a child to look through some multiplication examples,

3*2 = 6
4*3 = 12
...
...
10*1= 10

If the kid learns some general rules as to how x*y=z then he will be able to answer any new multiplication question I ask him. However, if he just memorizes the outcomes of the few examples that I showed them then they will get any new instance wrong. This is overfitting.

As another example, imagine fitting a 2nd order relationship between 2 variables with a 5th order polynomial. This will match the points too specifically and will not generalize to new data. Your validation error has stuck at 2.20, but training error is still decreasing. It may be the case of overfitting, because it occurs when it overlearn the data.

Try using L2 regularization, it overcome the overfitting problem, by forcing the weight parameters to be as small as possible.

Another thing, you can try using dropout technique, which will also help in interpreting model behavior.

Given the lack of detail, I would say that a possible reason is the use of dropout. Dropout is a regularization technique that nulls out randomly some elements during training, hence making the model behave worse with the training data but generalize better to unseen data.

Dropout is not applied when evaluating the model. This is one of the typical reasons of having lower validation loss than training loss.

This would explain with before step 150 you have better validation loss. At that step, your model has overfitted to the training data enough to compensate for the regularization and keeps overfitting, hence the training loss keeps improving while the validation one does not.