0
$\begingroup$

Good afternoon all,

I am working on a Deep Learning project where I'm proceeding in batches (rather than a continuous pipeline).

Essentially I've built a function that can convert raw data into vectors of magnitude ~800, and then I have around 3-5 Million of these vectors with a simple binary category: Pass/Fail.

In my ML projects, I found just storing these records in a CSV or Excel file was sufficient as the most data I've ever worked with was around 20,000 records, but now I don't believe either would suffice for my use case.

What would be the best way to store this data so that I can call an encoder at my next stage to run through the records?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.