I am working with the DecisionTreeRegressor and trying to understand how well the data fits the model. I calculated both RMSE and coefficient of determination. At a certain depth, coefficient of determination has the value equal to 0.9918744073066561 but the value of RMSE is equal to 75.0025. I cannot understand this. The value of RMSE is quite large but the value of the coefficient of determination is close to 1.0. What does it really mean? Is the model/fit good enough?

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.metrics import mean_squared_error

def rmse(y_true, y_pred):
    return math.sqrt(mean_squared_error(y_true, y_pred))

sample_depth = np.linspace(1,40, num = 40, dtype=int)
dt_score_list = []
for index, depth in enumerate(sample_depth):
    boosted_regressor = AdaBoostRegressor(DecisionTreeRegressor(max_depth=depth), random_state=1)
    boosted_regressor.fit(X_train, y_train)
    dt_score_list.append(boosted_regressor.score(X_test, y_test))
    print(boosted_regressor.score(X_test, y_test), rmse(y_test, boosted_regressor.predict(X_test)), depth)
  • $\begingroup$ RMSE is based entirely on your data, if you have a lot of data you can expect a lot of variation coming from a lot of points (which can be small but it does add up). $\endgroup$ – user2974951 Sep 25 '18 at 13:20

Compared to $R^2$ $RMSE$ is dependent on the variance of values. Even if $R^2$ is close to 1 but the standard deviation is high the value of $RMSE$ will also be high. $$RMSE=\sigma_{y_{true}}\sqrt{1-R^2}$$ where $\sigma$ is a standard deviation

As can be seen from the formula $RMSE$ and $R^2$ have a strict mathematical relationship that shows that $RMSE$ changes according to standard deviation ($\sigma$) and the higher the standard deviation of the true values the higher the value of $RMSE$.

$RMSE$ shows the deviation of the error in units of the values, while $R^2$ shows the share of variance that is explained by the model. You can explain 99% of the variance ($R^2$) but your numbers are in millions of dollars and your error still varies in thousands of dollars ($RMSE$) Whether thousands of dollars is ok for your error or not depends on the case. If you predict profit plus or minus 1000\$ is ok. If you predict revenue then plus or minus 1000\$ could be a difference between a profitable and a non-profitable company.

Proof of the formula:

According to sklearn the formula for $R^2$ is

$$R^2 = 1 - \frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}$$

RMSE is \begin{align} RMSE &= {\sqrt{\frac1n\sum_i^n(y_{true} - y_{pred})^2}} \\RMSE^2 &=\frac1n\sum_i^n(y_{true} - y_{pred})^2 \\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\frac1n\sum_i^n(y_{true} - y_{pred})^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} \\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2} \\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=1-\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2} \\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=R^2\\1-\frac{RMSE^2}{\sigma_{y_{true}}^2} &=R^2 \\RMSE&=\sigma_{y_{true}}\sqrt{1-R^2} \end{align}

where $\sigma$ is a standard deviation and $\overline{y}$ is an average


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.