Training a model sample by sample

Question

I'm training a Scikit model but it seems that in all examples, they call the fit method on the entire training set. What I want to do however is call it per sample (i.e. looping through all samples). This has multiple reasons but most importantly are

• MemoryError with my huge trainingset
• Training with new data instead of recompiling entire model

Yet when I loop and call the fit model self.gnb = self.gnb.fit(sample.data,labels) and then debug, the gnb model only has one class (namely the last one). So how should I approach this?

Answer 1

Not every model is able to learn sample-by-sample or incrementally. However, in scikit-learn, there're some models which have partial_fit method:

Incremental fit on a batch of samples.

This method is expected to be called several times consecutively on different chunks of a dataset so as to implement out-of-core or online learning.

This is especially useful when the whole dataset is too big to fit in memory at once.

This method has some performance and numerical stability overhead, hence it is better to call partial_fit on chunks of data that are as large as possible (as long as fitting in the memory budget) to hide the overhead.

You can just search for methods name in sklearn's documentation. This method exists, for example, for GaussianNB and Stohastic Gradient Descent, both Classifier and Regressor.

Also, you can use Random Forest and set number of samples (or sample ratio) per tree is small to fit the memory. Or use Dask and Dask ML to fit your data in memory.

Answer 2

In the general case this is impossible because standard learning algorithms require all the instances at once in order to compute the model. They usually work by making calculations based on the set of instances, then only the results of these calculations are stored as a model (for instance a linear regression classifier stores only the final weights which are needed to make a prediction).

It is however possible to train a model incrementally:

• Very often this is done by simply adding the new instances to the original training set, then re-train a new model on the full set of instances (new and old ones together).
• There are also actual incremental learning algorithms, including adaptations of some common learning methods. However this is not very common, I'm not aware of these being implemented in the most standard ML libraries that I know of.