I have an SQL DB which has millions of records(10 years of data) and wanted to implement some ML Models. Most of the courses/tutorials explains using a local file or downloaded CSV. But in my case I can't download all the records, but I need to load the data incrementally and train my model.How do i proceed about this?

And also if i'm wrong about incremental loading of data, how do companies load data for their ML models?.

I use Python language and am flexible with using libraries (scikit, tensorflow etc.)

Thank you.

  • 1
    $\begingroup$ Hi @SaiChandra, welcome to the site. Your question is quite broad, as the answer highly depends on the tooling available for each specific programming language and ML stack. It would be great if you could narrow it down to one or a few specific combinations of programming language and ML stack (e.g. python and scikit-learn). $\endgroup$
    – noe
    Apr 9, 2021 at 7:08
  • $\begingroup$ @noe My bad..sorry for th equestion to be a bit vague..for ML algorithms i use Python language with Pytorch, and I'm flexible in choosing a library. I'm struck with how the whole Data pipeline has to be built. Like how to load the data (huge data) into an algorithm..train it.. I thought I'll use schedulers to run SQL queries frequently and load data incrementally...but is this the right way? How do people build pipelines? Thank you. $\endgroup$ Apr 9, 2021 at 9:01
  • $\begingroup$ @noe I'm curious about Data pipelines because I cannot always download the entire data into my local systems and do ML operations. How do people in an organisation do that? $\endgroup$ Apr 9, 2021 at 9:11
  • $\begingroup$ Actually, as said there are plenty of options. The simplest one I can think is to train on batches of data, and update weights of your model batch by batch. $\endgroup$
    – Oscar
    Apr 9, 2021 at 9:23
  • $\begingroup$ @oscar..Yes batch loading is proper for this scenario, agreed. Also what are the other options that i can choose from? just for knowledge sake. $\endgroup$ Apr 9, 2021 at 9:33


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.