# Python sklearn model.predict() gives me different results depending on the amount of data [closed]

I train my XGBoostClassifier(). If my testing set has:

0: 100
1: 884

It attempts to predict 210 1's. Around 147 are wrong (False positives) and 63 1's correctly predicted (True positives).

Then I increase my testing sample:

0: 15,000
1: 884

It attempts to predict 56 1's. Around 40 are wrong (False positives) and 16 1's correctly predicted (True positives).

Am I missing something? some theory? some indication on how to use model.predict(X_test)?

Does it say somewhere - if you try to predict 10 items is gonna try harder than if you try to predict 10000 items? In what situation model.predict(X_test) would give me a different result for Joe Smith if his prediction is accompanied by 8000 more rows?

The code I use is the following:

from xgboost import XGBClassifier
xgb = XGBClassifier(subsample=0.75,scale_post_weight=30,min_child_weight=1,max_depth=3,gamma=5,colsample_bytree=0.75)
model = xgb.fit(X_train,y_train)
y_pred_output = model.predict(X_test)
cm = confusion_matrix(y_test, y_pred_output)

y_pred_output2 = model.predict(X_test2) #contains the same 884 1's plus 14500 more rows with 0's as the target value
cm = confusion_matrix(y_test2, y_pred_output2)

it produces two different matrices:

#Confusion matrix for y_test with 15000 0's and 884 1's
[[14864   136]
[  837    47]]

#Confusion matrix for y_test with 500 0's and 884 1's
[[459  41]
[681 203]]

Notice that the same 884 positive class items are being used across both attempts. Why would the true positives go down to 47 just because we now have more Negatives on the X_test?

• are you running .fit again between the two? Commented Oct 14, 2020 at 17:43
• No, I just edited the post to include the code. I fit the X_train, then do predict on the X_test. Commented Oct 14, 2020 at 17:52
• I saw that code. What I meant was, how are you getting the second test set, and what pieces of code are you running for that test set? Commented Oct 14, 2020 at 18:26
• From an external file. test.csv. It has 15000 0's and 884 1's... When I limit the 0's to the first 500 of them, my true positives go higher! I'm like: "why would you care how many 0's there are?" Commented Oct 14, 2020 at 19:13
• As already explained in the answer below, this should clearly not happen; but without the detailed way of your 2 test sets preparation, it is impossible to locate the exact issue - in all probability, this is a dataset preparation fault. These models actually perform the predictions one by one, so no single prediction "sees" (or cares about) the rest. Commented Oct 14, 2020 at 19:27