# SciKit-Learn Decision Tree Overfitting

I'm pursuing a computer science minor at my university, and one class I'm in is Machine Learning. We have a project to utilize a few algorithms we have learned so far. I've been using SciKit-Learn to perform these algorithms, but when it comes to decision trees I keep getting a feeling I am overfitting.

I'm using a dataset about weather, giving characteristics such as city, state, month, year, wind direction, wind speed, etc... where the target variable is the average temperature for the day. Now I know this is hard to classify, as it is pretty much a continuous variable space, but I've simplified it to the predicted being within a range of 5 to the actual.

The following is my code:

address2 = 'C:/.../weather.csv'

cityCode= le.fit_transform(weather.iloc[:,2])
windDirection = le.fit_transform(weather.iloc[:,3])
month = le.fit_transform(weather.iloc[:,8])
precip = le.fit_transform(weather.iloc[:,9])
windSpeed = le.fit_transform(weather.iloc[:,10])
state = le.fit_transform(weather.iloc[:,11])
week = le.fit_transform(weather.iloc[:,12])
year = le.fit_transform(weather.iloc[:,13])

Xweather = list(zip(cityCode,windDirection,month,precip,windSpeed,state,week,year))
yweather= weather.iloc[:,0]

yweather_test = train_test_split(Xweather, y, test_size = 0.2, random_state=413)

cWeather = tree.DecisionTreeClassifier()
cWeather.fit(Xweather_train,yweather_train)
accu_train_weather=np.sum(abs(cWeather.predict(Xweather_train)-yweather_train)<=5)/float(yweather_train.size)*100
accu_test_weather=np.sum(abs(cWeather.predict(Xweather_test)-yweather_test)<=5)/float(yweather_test.size)*100
print("Classificaton accuracy on training set", accu_train_weather, "%")
print("Classificaton accuracy on test set", accu_test_weather, "%")


My training set constantly gets 100% training accuracy, but the test set is constantly 57% accurate, which leads me to believe the tree is overfitting to the training set.

I know I'm not doing any pruning, but even when I do, I can get the same test accuracy as unpruned at best. By pruning I mean setting the tree classifier to have a maximum number of leaves, minimum samples per leaf, and maximum depth.

I'm not an advanced coder (as you can probably tell by my code), but any help would be great.

Predicting average temperature is a regression task, not classification. You should be using DecisionTreeRegressor instead. Temperature is a continuous value and you are treating it as a category by using a classifier.

Tinkering with the hyperparameters (maximum number of leaves, minimum samples per leaf, and maximum depth, etc) is still important since decision trees always are prone to overfitting. If you struggle to find good parameters yourself then you can try some automated methods such as GridSearchCV or RandomizedSearchCV in sklearn.

The vanilla decision tree algorithm is prone to overfitting. That's kind of why we have those ensembled tree algorithm. The classics include Random Forests, AdaBoost, and Gradient Boosted Trees. All of those are implemented in sklearn.

There are other more advanced variation/implementation outside sklearn, for example, lightGBM and xgboost etc.

If you must use the vanilla decision tree, trying to reduce the dimensionality of your inputs might help to reduce overfitting.

• Thanks for the answer. So even though I implement a rudimentary version of pruning (by limiting the number of leaves, minimum samples per leaf, and max depth) I can still overfit? – Paulfryy Feb 10 '19 at 15:21
• think this way, what would happen if you use the simplest possible decision tree there can be, the one with a depth of 1 and 2 leave node? Is it going to overfit? Not very likely isn't it? But the performance is also going to be miserable. The more complex your tree is, the greater the ability it has to model complex problem, but this complexity also means more likely to overfit. Some algorithms are better than the others because they can give you more expressive power but also least likely to overfit at the same time. – Louis T Feb 11 '19 at 6:59

It seems like the split is incorrect....

yweather_test = train_test_split(Xweather, y, test_size = 0.2, random_state=413)


Change it to:

xweather_train, xweather_test, yweather_train, yweather_test = train_test_split(
Xweather, yweather, test_size = 0.2, random_state=413)


Then do the prediction:

y_pred = classifier.predict(xweather_test)


Make the confusion matrix and see the result:

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(yweather_test, y_pred)


Note - I have just suggested some corrections based on just your code. Your data is not available on the link. I am assuming you have correctly converted the target variable from continuous to categorical range.