Skip to main content
3 of 3
added 270 characters in body

Update the saved model after training

Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text = chunk['body']
  label = chunk['user_id']
  print(text.shape, label.shape)

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3)
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

For example, if the first chunk had labels 1 and 2, and the second chunk 3 and 4, will the final model be able to predict just 3 and 4? Or 1 and 2 as well, given the testing data has all the labels. Any help?

UPDATE The chunk is used to get text from the csv. I have updated my code.