1
$\begingroup$

Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text = chunk['body']
  label = chunk['user_id']
  print(text.shape, label.shape)

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3)
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

For example, if the first chunk had labels 1 and 2, and the second chunk 3 and 4, will the final model be able to predict just 3 and 4? Or 1 and 2 as well, given the testing data has all the labels. Any help?

UPDATE The chunk is used to get text from the csv. I have updated my code.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ It is difficult to provide you with advise because your code is not clear. For example, chunk is not used inside the for loop - in fact, nothing is changing inside the loop. $\endgroup$
    – nwaldo
    Commented Apr 19, 2020 at 18:06
  • $\begingroup$ @nwaldo: Apologies. I have updated my code to make clear how chunk is being used inside the loop. Any answer would be helpful. $\endgroup$
    – Anan Srivastava
    Commented Apr 20, 2020 at 8:30

2 Answers 2

Reset to default
0
$\begingroup$

Short answer : a model will not be able to predict attribute it has not seen during the training process. From what I understand from your code, each model will erase the previous one.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. So how can I iteratively train the model (because the whole data set doesn't fit in the memory at once) and save the model trained on every chunk? Any answer would be helpful. I have updated my question for better clarity. $\endgroup$
    – Anan Srivastava
    Commented Apr 20, 2020 at 8:32
  • $\begingroup$ With multiple different models it will be difficult to know which one use to predict if you dont know if the model was trained with data similar as the one you want to predict $\endgroup$
    – Rusoiba
    Commented Apr 20, 2020 at 12:59
  • $\begingroup$ You could consider splitting the space into 4 or more subsets that groups similar data and then train a model for each of them. With the new observation to predict, you will have to determine in which subset it falls, so you will be able to ask the right model. If this answer helps, please consider voting up my answer :) $\endgroup$
    – Rusoiba
    Commented Apr 20, 2020 at 13:01
  • $\begingroup$ Thank you. I am not sure if you are quite understanding my question. Your solution will not scale as the data scales. In fact, it will take up more memory. So basically, there is no way to save a model trained on the whole data? $\endgroup$
    – Anan Srivastava
    Commented Apr 20, 2020 at 13:19
0
$\begingroup$

I'm assuming you do not have access to a larger machine (e.g. leveraging cloud compute) and you have to run this locally in some way. If that's the case, one suggestion I would make is to try a combination of Vaex and sklearn's SGDRegressor functionality.

You can think of Vaex as providing easy manipulation of dataframes which are larger than your memory (i.e. out-of-core Dataframes). With this, you can do your feature engineering and EDA in a more straightforward way than loading chunks.

When you are ready to model, you can leverage some of the models in sklearn which allow for training on batches of data at a time (e.g. SGDRegressor or SGDClassifier). These options provide a partial_fit method for training you model sequentially on different batches from your dataset.

Some links if you are interested in going down this path:

https://github.com/vaexio/vaex

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDRegressor.html

https://towardsdatascience.com/ml-impossible-train-a-1-billion-sample-model-in-20-minutes-with-vaex-and-scikit-learn-on-your-9e2968e6f385

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.