I am making a project on prediction cars price given its features. I was able to scrape over 13000 examples. After cleaning and manipulating the data, I left with a little more than 11000 examples, I used 12 features of car, like mileage, year, brand and all other important stuff. After model selection, I decided to use Random forests to predict model. There are several questions, that got me interested:
1) When tuning model, the best I can get on test set is mean absolute error more than 2200. R_2 score is 0.92 And training set error is 900-1000. R_2 score is 0.98 I couldn’t eliminate this overfitting, if it was, what I think it is. I tried grid search on bagging, forests even, on boosting (I know it reduces bias, not variance, but I was desperate) with different parameters, but the test error and train error with best parameters were always approximately same. I used simple estimators too, but they had too much bias in train and test. I know that there is a way to get rid off this overfitting, if only I had more data, but this data is maximum I can get from the page I scraped. So the first questions is: are there situations when overfitting cannot be solved without proper and big dataset? Am I stuck in this situation?
2) And the second optional question is: Some of the features in data are categorical, so after creating dummy variables I had something like 120 features. And interesting thing is, that after scaling, which I needed for PCA, the variance I retained with arbitrary n_components was completely different, than variance I retained from data scaled just on continuous features, and not on dummy. I know that there is no meaning in scaling dummy variables, and it’s not handful. But after scaling dummy variables, in order to retain 99 percent of variance, I needed something like 110 features (originally 120), and after scaling just continuous features, to retain 99 percent of variance I needed less than half of the features (~55). It’s strange for me, I cannot understand this behavior, is it okay at all?