# Decision tree and random forest over fitting

I am working on a real state data set to predict the price of buying a house in Dubai based on area, no.of bedrooms, number of baths and the town which the house is in.

All variables are numerical except the town variable which I transformed, using one hot encoding and target transform.

Then I applied linear regression, decision tree, random forest and I got the same results for both transforming methods.

Algorithm            Train score        Test score

linear regression    0.50               0.45
decision tree        0.93               0.79
random forest        0.94               0.77


From the results, I can read(but not so sure) that the data has information and has a high correlation scores but the model is over fitting. I used grid search to optimize the hyper parameters of the decision tree but the result did not improve.

So, the question is, what am I doing wrong?

Here are some questions that might help:

1. Do you have a balanced dataset? How does the distribution look like?
2. Is the distribution of the training and test set similar?
3. Did you try using cross validation?
4. What is your accuracy metric?
5. Are the features correlated among themselves?
6. I would also suggest using xgboost and compare its results.
• 3. I used cross_validate from sklearn and got a score around 0.5 but in sklearn there is no way to apply cross_validate on decision tree and pick a final model from it.(as i understood from multiple articles) – Mohammed Obeidat Nov 6 '19 at 8:27
• 4.The accuracy metric I used is R^2 – Mohammed Obeidat Nov 6 '19 at 8:27
• 5.The correlation for all features is between 0.5-0.9 except for the variable that resulted from applying target transform to the county variable which its correlation is around 0.1 – Mohammed Obeidat Nov 6 '19 at 8:31
• I was trying to suggest something like K Fold cross validation, you can take the mean of all the models rather than taking one. I would suggest using the mean absolute error (MAE) and RMSE to evaluate your results, that would allow you to identify which examples are not doing well on the model. How many country names do you have? Target transform might not be a good approach I would suggest looking at generating dummy variables for regression. Also you should try to visualize the country features and see if they make any sense as a predictor or just drop them. – Samarth Bhandari Nov 6 '19 at 16:27
• There are more than 100 different county names and they have some information because prices change significantly between different counties and when i excluded them the training result went down. I tried dummy variables but got the same results. – Mohammed Obeidat Nov 6 '19 at 18:03