Why am a getting wrong prediction when combining two list of samples, which individually gives correct prediction?

Question

So I am coding in Python. I have to set of samples. Set1 contains samples of class A and the other set, Set2 contains samples of class B. These samples taken are a part of the training dataset. When I am predicting set1 and set2 individually, the classification is perfect. Now when I am merging the two sets for prediction into one set, the prediction gives the wrong result for the samples in Set2, i.e., predicting the samples of set 2 to be in class A. However, samples belonging to Set1 are predicted to be in class A in the merged set. Why is this happening? (I have saved the model and loaded that model for further prediction, the same error persists)

model.add(Dense(newshape[1]+1, activation='relu', input_shape=(newshape[1],)))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(100, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(50, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
          optimizer='adam',
          metrics=['binary_accuracy'])
model.fit(X_train, y_train,validation_data=(X_test, y_test),validation_split=0.2, epochs=500, batch_size=25, verbose=0)

Answer 1

You have a mistake somewhere in either the data you provide as training data or the model you use for predicting.

If it was really the same model trained on the full data in all the cases, then any given instance in set2 would always be predicted with the same class (right or wrong), independently from the other instances in the test set. This implies that it's not the same model being applied when you predict individual sets or both merged.

As a test, save your model somewhere first then apply it to your different cases (the model that you saved, don't train it again).

---

Ok so you are trying to do 3 experiments:

• Training on full training set produces modelA, then apply modelA on set1
• Training on full training set produces modelB, then apply modelB on set2
• Training on full training set produces modelC, then apply modelC on both set1 and set2

But if this was really what you are doing the three models would be the same: modelA = modelB = modelC. Basically you could do this instead:

1. Training on full training set produces modelD
2. apply modelD on any set

Based on your description what probably happens is something like this:

• Training on instances of class A (error) produces modelA, then apply modelA on set1 -> perfect results
• Training on instances of class B (error) produces modelB, then apply modelB on set2 -> perfect results
• Training on instances of class A (error) produces modelC = modelA, then apply modelC on both set1 and set2: perfect for class A, completely wrong for class B

You need to check which instances you give as training set in each case.