python: why add new features of image to be worse result with xgboost

Question

there are about 93 rows of data with features, two classes label. And, there are about 49 one-hot value features, and there are about 10 features continuous value. I split the data randomly by train and valid data to predict it with following code:

model = XGBClassifier()
model.fit(X_train2, y_train2)
y_pred = model.predict(X_valid)
accuracy = accuracy_score(y_valid, y_pred)
print("Accuracy: %.2f%%" % (accuracy * 100.0))

I found that when I just used the one-hot value features, the accuracy is 78.95%, but when added another 10 features with continuous values, and the accuracy is reduced sharply, and is 57.89%.More, based on the feature importance, the ranking top 5 features is the continuous values features, also other continuous features have importance value. But most of the one-hot value feature importance value is 0, just 7 features have value. Of course, just use the one-hot value, there are just about 17 features have value, others are 0.

In addition, just used the 10 continuous features, the accuracy is 68.42%.

I don't know clearly why caused such kind of result. now I guessed that one of reason may be the number of data is not so big, and the features is almost about 59 features.

Answer 1

Guessing by your reported accuracies and the 93 sample points, it looks like you have just 19 samples in your validation set (the three accuracies you report then are 11/19, 13/19, and 15/19). So the smallest change in accuracy you can get is more than 5 percentage points, and your reported swings don't seem so drastic.

However, continuous variables can lead to overfitting, if your model learns each particular value separately (which would also lead to your continuous variables getting high feature importance scores). The default XGBClassifier parameter max_depth=3 makes that a little less likely, but the default n_estimators=100 for only 74 samples might well be at fault. I'd agree with bradS that you should tune hyperparameters a bit. Also, it's probably worth at least throwing something else like logistic regression at the data. But again I'm not sure how much fine-tuning you can realistically do with just 19 validation points.