After training a model on chunks, how can I save the final model?
df = pd.read_csv(, chunksize=10000) for chunk in df: text_clf.fit(X_train, y_train) filename = 'finalized_model.sav' joblib.dump(text_clf, filename) # load the model from disk loaded_model = joblib.load(filename)
Saving a model like this will just give me the model trained on the last chunk. How can I avoid that and get the overall model trained on every chunk?
UPDATE: Most of the real-world data sets are huge and can’t be trained in one go. How can I save a model after training it on each chunk of data?
df = pd.read_csv(“an.csv”, chunksize=6953) for chunk in df: text = chunk[‘body’] label = chunk[‘user_id’] X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3 ) text_clf = Pipeline([(‘vect’, TfidfVectorizer()), (‘tfidf’, TfidfTransformer()), (‘clf’, LinearSVC()), ]) text_clf.fit(X_train, y_train) # save the model to disk filename = ‘finalized_model.sav’ joblib.dump(model, filename)
Will saving it this way give me the model trained on the entire dataset? I want the model trained on every chunk. Any help?