As the title states, I am trying to cluster a huge dataset and cluster it by using sklearn.Birch to learn incrementally.
If it's a small dataset, I could just use gridsearchcv.
However, there's no built-in way to do that with large dataset in scikit-learn right now.
I'm curious if there is any suitable/general way to tune parameters batch by batch?
In Grid Search, Random Search, we trained different independent model i.e. irrespective of the outcome of previous models and later evaluate all these independent models.
We have another automatic technique for hyper-parameter optimization, known as Bayesian HyperOpt.
It took the references of previous model and try to identify the best new set of hyper param and trained a new model. This two step process gets repeated, until we get the desired results.
You can get many online references for complete implementation.
Although I could not find any approach for batch by batch, I thought its worth to mention Bayesian approach.
You can use the library for big data - dask, and in particular dask-ml. This library will allow you to work with large datasets that do not fit into memory by implementing batch learning. Dask-ml also has grid_search_cv.
