I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the timestamps are not synchronized). In total this means about 20k columns and hundreds of millions of rows. So far we have accumulated about 5TB of data on our local machine.
We would like to be able to train neural networks on this datasets which means we would need to be able to read random/non sequential individual rows or chunks of rows which are between 2 time indexes in to memory very fast.
I have never worked with mySQL but from reading about it, it seems it could be good for this task. I have however seen no examples of using mySQL with pytorch or pytorch Dataset. Is there a reason for this?
What is the conventional way that this task would be accomplished?