Overfitting problem in model

Question

I am making a project on prediction cars price given its features. I was able to scrape over 13000 examples. After cleaning and manipulating the data, I left with a little more than 11000 examples, I used 12 features of car, like mileage, year, brand and all other important stuff. After model selection, I decided to use Random forests to predict model. There are several questions, that got me interested:

1) When tuning model, the best I can get on test set is mean absolute error more than 2200. R_2 score is 0.92 And training set error is 900-1000. R_2 score is 0.98 I couldn’t eliminate this overfitting, if it was, what I think it is. I tried grid search on bagging, forests even, on boosting (I know it reduces bias, not variance, but I was desperate) with different parameters, but the test error and train error with best parameters were always approximately same. I used simple estimators too, but they had too much bias in train and test. I know that there is a way to get rid off this overfitting, if only I had more data, but this data is maximum I can get from the page I scraped. So the first questions is: are there situations when overfitting cannot be solved without proper and big dataset? Am I stuck in this situation?

2) And the second optional question is: Some of the features in data are categorical, so after creating dummy variables I had something like 120 features. And interesting thing is, that after scaling, which I needed for PCA, the variance I retained with arbitrary n_components was completely different, than variance I retained from data scaled just on continuous features, and not on dummy. I know that there is no meaning in scaling dummy variables, and it’s not handful. But after scaling dummy variables, in order to retain 99 percent of variance, I needed something like 110 features (originally 120), and after scaling just continuous features, to retain 99 percent of variance I needed less than half of the features (~55). It’s strange for me, I cannot understand this behavior, is it okay at all?

Answer 1

Overfitting is when the test score is way below the training score, and when the latter is high. Here your test score is 0.92 which is not bat at all, so according to this score there is no overfitting here. To see that, you can dedicate a part of your dataset to be a validation set. So that, you will be able to evaluate training score, test score and validation score.

In the same idea you can use cross-validation (if you use scikit here), which will split you dataset into k-folds, k-1 of them will be used for the training and 1 for the test. This process in then repeated so that each of the folds is used as the test set.

If the score varies at each iteration, it could possibly mean that you overfit, if not then you can say that your model is robust

For the second question what do you mean by continuous feature ?

Answer 2

I will reply only to the first one:

Yes, in all real problems overfit is unavoidable.

Every model can in principle lead to overfit. Overfitting is something that you can only limit, and never eliminate. This because given an arbitrary big number of parameters (trees, or iterations, in the case of Random Forest) you could obtain a ~100% match on the training set.

Now, in the hypothetical case where the training set includes all the samples existing in the universe, then you are OK with that model.

But as you know this is impossible.

So, in real life, where you have only a tiny sample of all the possible existing data, you have to limit your model in order to capture only the most important characteristic of your data, and left the details (which your model is unable to capture) behind.

The more details the model try to catch in the training dataset, the more is likely that they are pattern/structures which actually exist in the training set, but that do not hold true in general, and so the model is unable to behave well with unseen data.

As example: say you train the cat classifier with a dataset where a small percentage of the pictures show cats on sofa. If you go too deep in training it, the model can get to the conclusion that being on a sofa is a characteristic of the cats, which of course is wrong. While if you are more conservative, and you train the model less deeply, your model will ignore the detail "sofa", focusing on details that affect the majority of the cats, and this will lead your model to generalize better.

Here the amount of data helps: the more you have, the more those details become evident to a model. Following the example above, if in your dataset lots of pictures with different stuff/animals on sofas are present, it can give a better interpretation on things being on sofas, and so it can be trained deeply, or with more parameters, and so increase it's performance on the training set while keeping a good performance also on the test set.