Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I built a decision tree model and am not sure if it is good or bad. Could you help to evaluate my model?
My code:
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder()
X_new = encoder.fit_transform(X)
#Decision tree model
model = DecisionTreeRegressor(random_state=1)
# Fit model
model.fit(X_new, y)
print(model.predict(X_new))
Result that I got:
[ 2.86 12.83333333 4.5 ... 2.9 2.925
2.5 ]"
It appears that you are training your model and generating predictions on the same dataset (X_new). You should not attempt to evaluate your model's performance using this output - because you are applying the model to the same data you trained it on, your evaluation will be over-optimistic. You need to set a portion of your dataset aside as test data, train the model on the remainder, and then apply the model to the independent test data. You can then calculate performance metrics like MSE or MAE as mentioned in the other answers, which will give you an unbiased estimate of how your model performs on unseen data. Cross-validation is another way to get an unbiased estimate of performance, which is essentially repeating train-test splits in a way to leverage all the available data.
You should calculate some metric like MAE (mean absolute error) which calculates the average of the absolute difference between each prediction and the actual value. I recommend you kaggle's intro to machine learnig course where you do exactly this with decision trees and random forests
It's all about the bias-variance trade-off (https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). Simple models tend to have a high bias, but low variance and complex models tend to have the opposite effect. It's about searching for the right balance. I suggest you go and buy 'Elements of statistical learning', which is a superb reference for people who want to dive in ML.
Together with the no free lunch theorem, it's hard to determine whether your model is good or bad, except for trial and error. I usually use 2 metrics: the Weighted Mean Absolute Error (https://forums.fast.ai/t/how-to-calculate-weighted-mean-absolute-error-wmae/8575) for the bias component and a simple metric where I divide the MAPE(mean absolute percentage error) into buckets (how many predictions were close to 0% mape and how many were far worse). It gives you an idea of how the variance of a model is. A good visual representation of bias and variance is here:http://scott.fortmann-roe.com/docs/BiasVariance.html.
Another thing I use, is the Zero Rule algorithm to compare it to this one.
There is so much more to explain, so I would suggest to keep digging into articles, papers, sites like this,...!
A good model is the one devoid of underfitting or overfitting & which generalized well. To check whether your model is good or bad, you would have to plot your losses against parameters for both train and cross-validation data set. Parameter values, where differences among the losses of train and cv data are very close & also minimal, implies a good model with those parameter values.
