0
$\begingroup$

I built a decision tree model and am not sure if it is good or bad. Could you help to evaluate my model?

My code:

from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()
X_new = encoder.fit_transform(X)
#Decision tree model
model = DecisionTreeRegressor(random_state=1)

# Fit model
model.fit(X_new, y)
print(model.predict(X_new))

Result that I got:

[ 2.86       12.83333333  4.5        ...  2.9         2.925
  2.5       ]"
$\endgroup$
1
$\begingroup$

It appears that you are training your model and generating predictions on the same dataset (X_new). You should not attempt to evaluate your model's performance using this output - because you are applying the model to the same data you trained it on, your evaluation will be over-optimistic. You need to set a portion of your dataset aside as test data, train the model on the remainder, and then apply the model to the independent test data. You can then calculate performance metrics like MSE or MAE as mentioned in the other answers, which will give you an unbiased estimate of how your model performs on unseen data. Cross-validation is another way to get an unbiased estimate of performance, which is essentially repeating train-test splits in a way to leverage all the available data.

$\endgroup$
4
  • $\begingroup$ Yes, Because i have just one Data Set and it is Training Data set. Actually i am working on a project (Just for educational purpose) data downloaded from kaggle website and they have given two data sets train and test in two CSV files. Hence i used entire training set for this . If i have to cross validation How will i do it? Will it okay divide my training set again into training and test set? i am confused with two data sets already given by Kaggle. May you help me please? $\endgroup$
    – Amit Yadav
    Aug 17 '19 at 4:23
  • $\begingroup$ @AmitYadav To do K-fold cross validation, gather all the data you have, then randomly divide it into K groups as evenly as you can. Now you can repeat train/test K times, using every one of the K folds as the test data, while using the other K-1 folds as training data. $\endgroup$ Aug 19 '19 at 12:39
  • $\begingroup$ Understood now. Thanks Mr. Wang $\endgroup$
    – Amit Yadav
    Aug 19 '19 at 16:38
  • $\begingroup$ @AmitYadav In this case you should use your model on the provided test dataset by kaggle and calculate the kpis. $\endgroup$
    – Fnguyen
    Oct 16 '19 at 17:10
0
$\begingroup$

You should calculate some metric like MAE (mean absolute error) which calculates the average of the absolute difference between each prediction and the actual value. I recommend you kaggle's intro to machine learnig course where you do exactly this with decision trees and random forests

$\endgroup$
1
  • $\begingroup$ Thank you very much, Sure, Will sign up for the course too. $\endgroup$
    – Amit Yadav
    Aug 16 '19 at 10:02
0
$\begingroup$

It's all about the bias-variance trade-off (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). Simple models tend to have a high bias, but low variance and complex models tend to have the opposite effect. It's about searching for the right balance. I suggest you go and buy 'Elements of statistical learning', which is a superb reference for people who want to dive in ML.

Together with the no free lunch theorem, it's hard to determine whether your model is good or bad, except for trial and error. I usually use 2 metrics: the Weighted Mean Absolute Error (https://forums.fast.ai/t/how-to-calculate-weighted-mean-absolute-error-wmae/8575) for the bias component and a simple metric where I divide the MAPE(mean absolute percentage error) into buckets (how many predictions were close to 0% mape and how many were far worse). It gives you an idea of how the variance of a model is. A good visual representation of bias and variance is here:http://scott.fortmann-roe.com/docs/BiasVariance.html.

Another thing I use, is the Zero Rule algorithm to compare it to this one.

There is so much more to explain, so I would suggest to keep digging into articles, papers, sites like this,...!

$\endgroup$
0
$\begingroup$

A good model is the one devoid of underfitting or overfitting & which generalized well. To check whether your model is good or bad, you would have to plot your losses against parameters for both train and cross-validation data set. Parameter values, where differences among the losses of train and cv data are very close & also minimal, implies a good model with those parameter values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.