3
$\begingroup$

I have big table in dataframe (600k rows) which has y column (the variable I want to predict) and other 4 other columns that are the X. I have run RF regressor and I got score of 0.87 when I run it on the train and test.

However, when I tried to predict another set of data (which is very similar, with 1M rows) I got score of 0.65. So I assumed that is overfitting. when I tried to understand why it hapenns, I went back to the distribution of the y column, which looks like this:

enter image description here

my question is, can it be that because that my data does not have normal distribution (or very skewd...) my model preformance is bad? Do all variables need to have normal distribution? how does the score of the random forest regrssion is calculated? id value is 0.25 and predict is 0.26 does it count as correct prediction?

Improve this question
$\endgroup$
1
  • $\begingroup$ how can a machine learning model have identical train and test scores? by this do you mean you predicted the train set and predicted the test set, and both came out 0.87? $\endgroup$
    – develarist
    Commented Nov 8, 2020 at 11:21

1 Answer 1

Reset to default
2
$\begingroup$

If you use tree-based algorithms like random forests the data distribution should not be an issue. Linear algorithms are more dependent on the distribution of your variables. To check if you overfit can try to predict your training data and compare the result with test data. The score depends on your evaluation metric. If you use scikit-learn you get R^2 as your metric.

The coefficient R^2 is defined as (1 - u/v), where u is the residual sum of squares ((y_true - y_pred) ** 2).sum() and v is the total sum of squares ((y_true - y_true.mean()) ** 2).sum().

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Even in linear regression, the normality assumption applies to the conditional distribution of the response variable, not the marginal (pooled) distribution of the response variable. Further, when we make that assumption of normality, it is for inference, not so much for prediction. In particular, the Gauss-Markov theorem makes no assumption about a normal conditional distribution. $\endgroup$
    – Dave
    Commented Oct 29, 2020 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.