After training a model on chunks, how can I save the final model?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

Saving a model like this will just give me the model trained on the last chunk. How can I avoid that and get the overall model trained on every chunk?

UPDATE: Most of the real-world data sets are huge and can’t be trained in one go. How can I save a model after training it on each chunk of data?

df = pd.read_csv(“an.csv”, chunksize=6953)
for chunk in df:
  text = chunk[‘body’]
  label = chunk[‘user_id’]

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3 )

  text_clf = Pipeline([(‘vect’, TfidfVectorizer()),
  (‘tfidf’, TfidfTransformer()),
  (‘clf’, LinearSVC()),

  text_clf.fit(X_train, y_train)

  # save the model to disk
  filename = ‘finalized_model.sav’
  joblib.dump(model, filename)

Will saving it this way give me the model trained on the entire dataset? I want the model trained on every chunk. Any help?


One option is Dask framework's incremental learning. That would automate the loading of chunks and applying partial_fit in scikit-learn.

The code would be something like:

from dask_ml.datasets import make_classification
from dask_ml.wrappers import Incremental

X, y = make_classification(chunks=25)
estimator = LinearSVC(random_state=10, max_iter=100)
clf = Incremental(estimator)
clf.fit(X, y, classes=[0, 1])

joblib.dump(clf, filename)

It seems that you are using 'fit' method, which basically overwrites the previous training and trains a new model, on your new chunk; that is why you are not able to save the model. You can think of this problem as an 'incremental training' problem, and what you basically require is to partially fit using partial_fit() method. Make sure that your classification algorithm supports partial_fit()

  • $\begingroup$ Thank you for your response. For classifiers that don't support partial fit, what is a way to get past it? $\endgroup$ – Anan Srivastava Apr 15 '20 at 8:46
  • $\begingroup$ I guess the only way to achieve a partial fit-like results without using partial_fit() or not having partial_fit is to write your own code which trains the model, stores the parameters, and updates them after every training. Moreover, you can use other frameworks, like Dask. $\endgroup$ – Danish Shakeel Apr 15 '20 at 8:48
  • $\begingroup$ Thank you. Is there a pseudo code that you can kindly provide? $\endgroup$ – Anan Srivastava Apr 15 '20 at 8:49
  • $\begingroup$ I tried looking up on the internet for it, however I don't find any. You can try other frameworks, like Dask (dask-ml.readthedocs.io/en/latest/incremental.html) $\endgroup$ – Danish Shakeel Apr 15 '20 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.