I need to know how would I get to know if I have overfitted my Machine Learning model on the train data. The performance metric I have used is Logistic Loss. Does the stability of the features affect the performance of my model? If yes, how do they relate?
You need to look for the differences in the training loss and the cross-validation & test losses. If those are low, it means the model performs fairly well. Ideally, train loss should be roughly equal to the cross-validation and test losses. If not, the model is overfitted.
This difference also hints at an insignificant overlap between the train data points and the cross-validation and test data points. Such features are said to be unstable. The model in such a case only gets to see the data points in the train data and not those in the cross-validation and test data, and is simply overfitted thereby. It, hence, performs poorly. You can verify the same by computing the percentage of data points present in the cross-validation and test data out of those in the train data for different features in your dataset.
-
$\begingroup$ It's not that simple; a large difference between train & test loss may be a sign of underfitting, too. $\endgroup$ – desertnaut Oct 13 '20 at 15:51
-
2$\begingroup$ @desertnaut Yeah, you got that right. But it ain't that difficult as well! $\endgroup$ – Shayan Shafiq Oct 14 '20 at 4:31