2
$\begingroup$

I am trying an exercise where I have been asked to "Evaluate each model accuracy on testing data set for a max_depth parameter value changing from 2 to 5".

The model here is DecisionTreeRegressor. I just wanted to know what is the metric for calculating the Accuracy for a DecisionTreeRegressor model.

My understanding is that it's same as Score which can be calculated simply as regressor.score(X_test, Y_test)

Please let me know what should be used to calculate the Accuracy of the DecisionTreeRegressor Model.

$\endgroup$
2
+25
$\begingroup$

Accuracy in ML vocabulary is used mostly for Classification problem i.e. Count of correct prediction out of total.
In a common speaking language, it will mean the predictive correctness of the model esp. on test data.

My understanding is that it's same as Score which can be calculated simply as
regressor.score(X_test, Y_test)

I am assuming that you are using SciKit-Learn, score method for DecisionTreeRegressor will return R-square coefficient.Offical Link

score(self, X, y[, sample_weight])
Return the coefficient of determination R^2 of the prediction.

What should you do -
You should calculate two metrics - R-square and MAE/MSE.
Reason being - for an end-user/business person, MAE would be useful e.g. saying that model's prediction will be ~250$ away from the correct value on an average.

Challenge with MAE/MSE is that it doesn't say if it is good model unless you have an idea of the underlying data. e.g. Creating two models on pricing data of 2 different city - Boston/Tokyo and the MSE is 1000$/$1500.
You can't conclude that the former is a better model from this data.
R-square helps here.

Adjusted R-square (Another regression metrics) - If your feature set is fixed, then you need not check this metrics. It was devised to fix an issue with R-square when the feature set is different for different models.


Snippet to get RMSE, R-square, Adjusted R-square

#https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics

def reg_metrics(y_test, y_pred, X_train):
    from sklearn.metrics import mean_squared_error, r2_score 

    rmse = np.sqrt(mean_squared_error(y_test,y_pred))
    r2 = r2_score(y_test,y_pred)

    # Scikit-learn doesn't have adjusted r-square, hence custom code
    n = y_pred.shape[0]
    k = X_train.shape[1]
    adj_r_sq = 1 - (1 - r2)*(n-1)/(n-1-k)

    print(rmse, r2, adj_r_sq)

Links to study -
Statistics by Jim
Wikipedia

| improve this answer | |
$\endgroup$
0
$\begingroup$

Majorly 3 types of machine learning model are present clustering, classification and regression. Each of them have different way of calculating accuracy. In case of regression following are the metrics available in scikit learn package. For more metrics check this LINK

mean_absolute_error

mean_squared_error

mean_squared_log_error

median_absolute_error

r2_score

| improve this answer | |
$\endgroup$
0
$\begingroup$

In the regression case, the best option would be to use $R^2$ or Adjusted $R^2$.

Also, you can use Other techniques to see if the model is behaving sensibly are related to the analysis of the residuals (the difference between actual y-value and predicted y-value) of the data points used to build the model such as Mean Squared Error(MSE), Root-Mean-Squared-Error(RMSE), and Mean-Absolute-Error(MAE).

| improve this answer | |
$\endgroup$
0
$\begingroup$

There is a way to measure the accuracy of a regression task. That is to transform it into a classification task.

The first approach is to make the model output prediction interval instead of a number. This is especially possible with decision trees, but it's better to use Quantile Decision Trees. Then you could have, say, a 95% prediction interval for each output of the model and calculate the accuracy by treating the true y-values that are inside the prediction intervals as a correct prediction. You could use this library for Quantile Trees.

The second approach is to divide the output range into intervals by hand and to treat those intervals as classes. You could choose the interval boundaries at the points of the lowest distribution of y-values. Or use a fuzzy classification boundary, where fuzzy interval around the boundary is considered to be true for both neighboring classes.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.