The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass operations that I wrote in the future. If I read data line by line, is there any way I can still use sklearn pipeline API?

  • $\begingroup$ Maybe not answering your question, but if you are doing nlp, I have found gensim's support of disk-based corpus reading to really help reduce memory usage and allow big models to be trained on small compute $\endgroup$
    – Ken Syme
    Feb 8 at 12:15

Sklearn has some strategies on its website. Mainly, some of their estimators allow incremental learning, meaning that you can feed the data in streaming fashion during training.

To use it with a pipeline, I would refer you to this StackOverflow post. You essentially have to break down the pipeline during training, and then during inference, you can use it as is.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.