I try to classify data from a dataset of 315 lines and 17 (real data) features (315x17). The target value is either "good" or "bad" (binary classification).
I used XGBoost to classify these data, but I get to much overfitting. I do cross validation using the logloss to evaluate the performance.
Green : logloss validation curve
Red : logloss training curve
X-axis : nrounds (max number of boosting iterations)
The first model (first picture was generated with :
bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.1,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 4,
# maximum depth of a tree
max_depth = 14,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 0,
eval_metric = "logloss",
nrounds = 2000,
verbose = TRUE,
subsample = 0.7,
print_every_n = 10,
early_stop_round = 10)
Then, I tried (second picture) :
bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.01,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 4,
# maximum depth of a tree
max_depth = 6,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 1,
eval_metric = "logloss",
nrounds = 2000,
verbose = TRUE,
subsample = 0.7,
print_every_n = 10,
early_stop_round = 10)
Basically, it is possible to reduce overfitting by changing max_depth, min_child_weight , gamma, subsample.
Below, again, I reduce the overfitting (3rd picture) :
bst.res <- xgb.cv(nfold = 4,
data = matrix_training_set,
label = training_set_label,
eta = 0.01,
# minimum sum of instance weight(hessian) needed in a child
min_child_weight = 7,
# maximum depth of a tree
max_depth = 6,
objective = "binary:logistic",
#L2 parameter
lambda = 1.1,
# L1 parameter
alpha = 0.5,
gamma = 5,
eval_metric = "logloss",
nrounds = 2000,
colsample_bytree = 0.7,
verbose = TRUE,
subsample = 0.3,
print_every_n = 10,
early_stop_round = 10)
Now it seems that the model is biased (red and green are indeed really close, take care of the scale).
I also tried grid searching, but the validation curve always stand really high (over 0.5 for logloss and less than 0.62 for AUC).
Now I'm wondering if I should create a model more biased (make the trees simplier), and then add more data to reduce this bias.
Any ideas to make this green curve lower ? Is that possible that there is no possible correlation between my target ("good" or "bad") and my features, which means that it becomes impossible to create a classifier from these data ?