I am trying to train XGBOOST model.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=43, stratify=y)

when I'm using train_test_split and pass the model X_train, Y_train and for eval_set X_test, Y_test, The model seems to be a very good one.

CM example:

enter image description here

But when I manually split the Dataset :

splitValidationIndex = round(dataset.shape[0]*0.6)
splitTestIndex = round(dataset.shape[0]*0.8)
X_train = X[:splitValidationIndex]
y_train = y[:splitValidationIndex]

Pass it to fit

X_val = X[splitValidationIndex:splitTestIndex]
y_val = y[splitValidationIndex:splitTestIndex]

Pass it to eval_set

X_test = X[splitTestIndex:]
y_test = y[splitTestIndex:]

Check the model prediction on that

that produced a much worse model


enter image description here

What am I missing/doing wrong?

  • $\begingroup$ I'm curious, what do you use to produce the multiple-threshold CMs? (Custom code, or a package? How do you determine which thresholds to print results for?) $\endgroup$ – Ben Reiniger Feb 27 '20 at 14:53
  • $\begingroup$ its some code we wrote. $\endgroup$ – Amit Raz Mar 5 '20 at 7:26

All looks correct, but you have to get to know about some details.

train_test_split splits arrays or matrices into a random train and test subsets. If you not specifying random_state, you will get a different result, it means that every time when you run your code train and test datasets would have different values each time.

If you fixed specific value like random_state = 42 then yours data in test and train set conserve the same values.

So in the first way splitting, model is learning on the specific(always the same) chunk of data, which could give a good results, in second way every time model is learning on different chunks of data, so results could be worse(by the way, its looks like overfitting on chunk(in first way of split).

In general no matter what way you choose, but you have to use cross-validation to create a good performing model.


Three things to check:

  1. train_test_split by default randomly splits the dataset, while your manual split splits into contiguous chunks of indices. If your data is not shuffled and there is some meaning to the index order (e.g. if they are time-ordered, and then especially if there is concept drift in your data), then these can produce very different results.

  2. You have set stratify=y, so train_test_split will split the dataset to have roughly equal proportions of each class in each fold. As above, if your dataset has a higher concentration of one class early compared to late, this will affect the testing significantly.

  3. Maybe your data and/or model is fairly unstable, so that different splits just produce rather different results; in this case, rerunning everything after a new train_test_split (without specifying the seed) should give you rather different results too.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.