Good afternoon all,
I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).
Essentially I've built a function that can convert raw data into vectors of magnitude ~800, and then I have around 3-5 Million of these vectors with a simple binary category: Pass/Fail.
In my ML projects, I found just storing these records in a CSV or Excel file was sufficient as the most data I've ever worked with was around 20,000 records, but now I don't believe either would suffice for my use case.
What would be the best way to store this data so that I can call an encoder at my next stage to run through the records?