# python: why add new features of image to be worse result with xgboost

there are about 93 rows of data with features, two classes label. And, there are about 49 one-hot value features, and there are about 10 features continuous value. I split the data randomly by train and valid data to predict it with following code:

model = XGBClassifier()
model.fit(X_train2, y_train2)
y_pred = model.predict(X_valid)
accuracy = accuracy_score(y_valid, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))


I found that when I just used the one-hot value features, the accuracy is 78.95%, but when added another 10 features with continuous values, and the accuracy is reduced sharply, and is 57.89%.More, based on the feature importance, the ranking top 5 features is the continuous values features, also other continuous features have importance value. But most of the one-hot value feature importance value is 0, just 7 features have value. Of course, just use the one-hot value, there are just about 17 features have value, others are 0.

In addition, just used the 10 continuous features, the accuracy is 68.42%.

I don't know clearly why caused such kind of result. now I guessed that one of reason may be the number of data is not so big, and the features is almost about 59 features.

• You are effectively using different datasets each time. Have you re-tuned the xgboost parameters each time you run the "new" data through the model? That might help. – bradS Jan 29 '19 at 10:09
• I don't know add the new features , the accuracy difference is so big. I just used the default parameters not re-tuned. – tktktk0711 Jan 29 '19 at 10:22

However, continuous variables can lead to overfitting, if your model learns each particular value separately (which would also lead to your continuous variables getting high feature importance scores). The default XGBClassifier parameter max_depth=3 makes that a little less likely, but the default n_estimators=100 for only 74 samples might well be at fault. I'd agree with bradS that you should tune hyperparameters a bit. Also, it's probably worth at least throwing something else like logistic regression at the data. But again I'm not sure how much fine-tuning you can realistically do with just 19 validation points.