The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass operations that I wrote in the future. If I read data line by line, is there any way I can still use sklearn pipeline API?
1
-
$\begingroup$ Maybe not answering your question, but if you are doing nlp, I have found gensim's support of disk-based corpus reading to really help reduce memory usage and allow big models to be trained on small compute $\endgroup$– Ken SymeFeb 8, 2021 at 12:15
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Dask can help you out. Basically it uses sparse data to load your dataset so that even datasets much larger than your compute memory can be loaded.
$\begingroup$
$\endgroup$
Sklearn has some strategies on its website. Mainly, some of their estimators allow incremental learning, meaning that you can feed the data in streaming fashion during training.
To use it with a pipeline, I would refer you to this StackOverflow post. You essentially have to break down the pipeline during training, and then during inference, you can use it as is.
answered May 6, 2020 at 12:28