Good afternoon all,

I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).

Essentially I've built a function that can convert raw data into vectors of magnitude ~800, and then I have around 3-5 Million of these vectors with a simple binary category: Pass/Fail.

In my ML projects, I found just storing these records in a CSV or Excel file was sufficient as the most data I've ever worked with was around 20,000 records, but now I don't believe either would suffice for my use case.

What would be the best way to store this data so that I can call an encoder at my next stage to run through the records?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.