Coefficient of determination is close to 1 but the value of RMSE is large. What does it mean?

Question

I am working with the DecisionTreeRegressor and trying to understand how well the data fits the model. I calculated both RMSE and coefficient of determination. At a certain depth, coefficient of determination has the value equal to 0.9918744073066561 but the value of RMSE is equal to 75.0025. I cannot understand this. The value of RMSE is quite large but the value of the coefficient of determination is close to 1.0. What does it really mean? Is the model/fit good enough?

from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import AdaBoostRegressor
from sklearn.metrics import mean_squared_error


def rmse(y_true, y_pred):
    return math.sqrt(mean_squared_error(y_true, y_pred))


sample_depth = np.linspace(1,40, num = 40, dtype=int)
dt_score_list = []
for index, depth in enumerate(sample_depth):
    boosted_regressor = AdaBoostRegressor(DecisionTreeRegressor(max_depth=depth), random_state=1)
    boosted_regressor.fit(X_train, y_train)
    dt_score_list.append(boosted_regressor.score(X_test, y_test))
    print(boosted_regressor.score(X_test, y_test), rmse(y_test, boosted_regressor.predict(X_test)), depth)

Answer 1

Compared to $R^2$ $RMSE$ is dependent on the variance of values. Even if $R^2$ is close to 1 but the standard deviation is high the value of $RMSE$ will also be high. $$RMSE=\sigma_{y_{true}}\sqrt{1-R^2}$$ where $\sigma$ is a standard deviation

As can be seen from the formula $RMSE$ and $R^2$ have a strict mathematical relationship that shows that $RMSE$ changes according to standard deviation ($\sigma$) and the higher the standard deviation of the true values the higher the value of $RMSE$.

$RMSE$ shows the deviation of the error in units of the values, while $R^2$ shows the share of variance that is explained by the model. You can explain 99% of the variance ($R^2$) but your numbers are in millions of dollars and your error still varies in thousands of dollars ($RMSE$) Whether thousands of dollars is ok for your error or not depends on the case. If you predict profit plus or minus 1000\$ is ok. If you predict revenue then plus or minus 1000\$ could be a difference between a profitable and a non-profitable company.

Proof of the formula:

According to sklearn the formula for $R^2$ is

$$R^2 = 1 - \frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2}$$

RMSE is \begin{align} RMSE &= {\sqrt{\frac1n\sum_i^n(y_{true} - y_{pred})^2}} \\RMSE^2 &=\frac1n\sum_i^n(y_{true} - y_{pred})^2 \\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\frac1n\sum_i^n(y_{true} - y_{pred})^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} \\\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2} \\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=1-\frac{\sum_i^n(y_{true} - y_{pred})^2}{\sum_i^n(y_{true} - \overline{y}_{true})^2} \\1-\frac{RMSE^2}{\frac1n\sum_i^n(y_{true} - \overline{y}_{true})^2} &=R^2\\1-\frac{RMSE^2}{\sigma_{y_{true}}^2} &=R^2 \\RMSE&=\sigma_{y_{true}}\sqrt{1-R^2} \end{align}

where $\sigma$ is a standard deviation and $\overline{y}$ is an average