The problem I'm facing is that my data is too big, i can't load it to a dataframe and then process it. However, I really want to use the sklearn pipeline API, so that I can reuse those subclass operations that I wrote in the future. If I read data line by line, is there any way I can still use sklearn pipeline API?
Sklearn has some strategies on its website. Mainly, some of their estimators allow incremental learning, meaning that you can feed the data in streaming fashion during training.
To use it with a pipeline, I would refer you to this StackOverflow post. You essentially have to break down the pipeline during training, and then during inference, you can use it as is.