1
$\begingroup$

I train my XGBoostClassifier(). If my testing set has:

0: 100 
1: 884

It attempts to predict 210 1's. Around 147 are wrong (False positives) and 63 1's correctly predicted (True positives).

Then I increase my testing sample:

0: 15,000
1: 884

It attempts to predict 56 1's. Around 40 are wrong (False positives) and 16 1's correctly predicted (True positives).

Am I missing something? some theory? some indication on how to use model.predict(X_test)?

Does it say somewhere - if you try to predict 10 items is gonna try harder than if you try to predict 10000 items? In what situation model.predict(X_test) would give me a different result for Joe Smith if his prediction is accompanied by 8000 more rows?

The code I use is the following:

from xgboost import XGBClassifier
xgb = XGBClassifier(subsample=0.75,scale_post_weight=30,min_child_weight=1,max_depth=3,gamma=5,colsample_bytree=0.75)
model = xgb.fit(X_train,y_train)
y_pred_output = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred_output)

y_pred_output2 = model.predict(X_test2) #contains the same 884 1's plus 14500 more rows with 0's as the target value
cm = confusion_matrix(y_test2, y_pred_output2)

it produces two different matrices:

#Confusion matrix for y_test with 15000 0's and 884 1's
[[14864   136]
 [  837    47]]

#Confusion matrix for y_test with 500 0's and 884 1's
[[459  41]
 [681 203]]

Notice that the same 884 positive class items are being used across both attempts. Why would the true positives go down to 47 just because we now have more Negatives on the X_test?

$\endgroup$
7
  • $\begingroup$ are you running .fit again between the two? $\endgroup$
    – Ben Reiniger
    Oct 14, 2020 at 17:43
  • $\begingroup$ No, I just edited the post to include the code. I fit the X_train, then do predict on the X_test. $\endgroup$
    – morpho4444
    Oct 14, 2020 at 17:52
  • $\begingroup$ I saw that code. What I meant was, how are you getting the second test set, and what pieces of code are you running for that test set? $\endgroup$
    – Ben Reiniger
    Oct 14, 2020 at 18:26
  • $\begingroup$ From an external file. test.csv. It has 15000 0's and 884 1's... When I limit the 0's to the first 500 of them, my true positives go higher! I'm like: "why would you care how many 0's there are?" $\endgroup$
    – morpho4444
    Oct 14, 2020 at 19:13
  • $\begingroup$ As already explained in the answer below, this should clearly not happen; but without the detailed way of your 2 test sets preparation, it is impossible to locate the exact issue - in all probability, this is a dataset preparation fault. These models actually perform the predictions one by one, so no single prediction "sees" (or cares about) the rest. $\endgroup$
    – desertnaut
    Oct 14, 2020 at 19:27

1 Answer 1

2
$\begingroup$

If XGBoostClassifier is fed the same input data over and over again it will yield the same results. There is no inherent randomness in this classifier that would different results for the same input. Additionally - there should be no difference in the result of an individual prediction if it's requested in a smaller batch versus a larger batch (again the result will be identical).

On the other hand - if you train XGBoost on different data their outputs will definitely be different. If you add new data to the underlying dataset & train with that - new & different patterns will emerge that XGBoost will try to take advantage of and the entire tree network will be fit very differently.

I suspect what you are observing is a bug in structuring your input data that you are then feeding to the .predict() method. If you share a sample of your code maybe we can drill-down on the issue.

$\endgroup$
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.