1
$\begingroup$

I trained a xgboost classifier and it gave an accuracy of 49.99 % and i saved that model into a pickle file. When i ran the same data with pickle file (.pkl) it's giving an accuracy of 88.99 percent. I don't know why it's happening. Please help me out from this situation.

bank_dataset = pd.read_csv(r"dataset.csv")

missing_val = pd.DataFrame(bank_dataset.isnull().sum())

bank_dataset[' Balance'] = bank_dataset[' Balance'].fillna(bank_dataset[' Balance'].mean())

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

correlation = bank_dataset.corr()
print(correlation['Outcome'].sort_values(ascending = False), '\n')

k = 10
cols = correlation.nlargest(k, 'Outcome')['Outcome'].index
print(cols)
cm = np.corrcoef(bank_dataset[cols].values.T)
f, ax = plt.subplots(figsize=(14,14))
sns.heatmap(cm, vmax = .8, linewidths = 0.01, square = True, annot = True, cmap = "coolwarm", linecolor = "white", 
           annot_kws = {'size':12}, xticklabels = cols.values, yticklabels = cols.values)

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)
  
from xgboost import XGBClassifier

xg = XGBClassifier()
xg.fit(X_train, y_train)

y_pred = xg.predict(X_test)

from sklearn.metrics import confusion_matrix
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred))
cm = confusion_matrix(y_test, y_pred)

import pickle

filename = 'xg.pkl'
pickle.dump(xg, open(filename, 'wb'))

The above code for training and saving the model into .pkl

bank_dataset = pd.read_csv(r"dataset.csv")

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()

objList = bank_dataset.select_dtypes(include = "object").columns

for feat in objList:
    bank_dataset[feat] = le.fit_transform(bank_dataset[feat].astype(str))

X = bank_dataset.iloc[:, [7,8,12,24,11,16,4,18,20]].values
y = bank_dataset.iloc[:, -4].values

file = open('xg.pkl', 'rb')

data = pickle.load(file)

X_ = data.predict(X)

The second code is running the code with pkl file with the same data used for training.

$\endgroup$
  • $\begingroup$ You have an error somewhere: either it's not the same test data or it's not the same model. Note that a difference in the training data could cause a different model, especially in case the test data is included by mistake in the trainiing data in the second case. $\endgroup$ – Erwan Sep 14 at 18:14
  • $\begingroup$ i think you are making a mistake somewhere , can you please post the code. $\endgroup$ – Madhur Yadav Sep 14 at 18:15
  • $\begingroup$ Yes, I've edited the post and added two codes one for training the model and one using pkl file for predictions. Both codes use the same data $\endgroup$ – Nithin Reddy Sep 14 at 19:22
2
$\begingroup$

In the first code you split the data X randomly with this line:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

Then after training the model you correctly apply it to the test set, which is 20% of the instances:

y_pred = xg.predict(X_test)

Whereas in the second code you apply the model to the full data X instead of only to the test set:

X_ = data.predict(X)

So in this second code you are testing on the whole dataset which includes the 80% training set. It is of course incorrect to test on the training set, this is why you obtain artificially high performance: the model "is cheating" since it has seen 80% of the instances during training.


[answer to comment]

There are at least two possibilities to split the data properly and use the saved model only on the test data:

  • option 1: split the data before running the training code into two different files, one for the training set and one for the test set. The training code should be provided only the training data as input, it doesn't perform any splitting and just outputs the model. The testing code takes the model and the test data as input and outputs the predictions (or performance).
  • option 2: apply exactly the same splitting in the testing code as in the training code. This can be done thanks to the random_state parameter in the call to train_test_split: by using the same value for this parameter the function will always separate the data exactly the same way. Then of course use only the test set for predicting and evaluating (ignore the training set part of the data).

Option 2 is much easier to implement given your current code (you just have to copy the train_test_split call). Option 1 is more general and makes the ML design cleaner, but it's less convenient.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Ok, I understood then how to train total data without splitting and get the predictions and accuracy? I mean to train the total data and save that into pkl $\endgroup$ – Nithin Reddy Sep 15 at 5:12
  • $\begingroup$ Your question is not clear, I assume you meant how to properly separate training and test set when using a saved model, see edited answer. $\endgroup$ – Erwan Sep 15 at 10:37
  • $\begingroup$ No, the question is when i train with 80% and test with 20% the accuracy is 49 and when i train the model without splitting the data it's 98%. How come this happens? $\endgroup$ – Nithin Reddy Sep 15 at 18:08
  • $\begingroup$ As I said in the answer, that's because you're applying the model to the instances that it has seen during training so it knows the answer already. It's like a student who is given all the answers before the exam, they might get a very good grade but it's cheating. $\endgroup$ – Erwan Sep 15 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.