How create a representative small subset from a huge dataset, for local development?

Question

​

I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests and some very streamlined "analytic" tests pass, my code is not a mess, and my model is actually trying to do something meaningful with this data. I know that I cannot create a very good "representative" small subset which can totally mimick the original, but I want to make sure I find many of my model's base flaws with this data before training it on that huge dataset. Maybe having multiple different sized batches for different scopes of tests is an option too, I don't have any preferences.

​

What is the best strategy to create this subset? I think for a data that is not sequential, unlike mine, random downsampling of the datapoints might be a good thing, but I don't know what is a good practice in time Series data. Should I just choose a small frame of time as the new dataset? What about casuality? How to sample according to class imbalance? These are the first questions that come to my mind. But feel free to expand on even more questions.

Edit: What I am working on is this dataset. The dataset is quite large, and I want to effectively choose a subset from it. The task is to detect seizures. One option is to the number of subjects, I think. But I am open to all options that you might suggest!

Answer 1

In your case, you have first to deal with the biological data complexity.

I don't know the minimum sampling rate to detect brain epilepsia or any brain behavior. I would recommend to study some articles to know the best practices about EEG signal analysis like this one : https://www.frontiersin.org/articles/10.3389/fneur.2020.00375/full

Maybe there are good practices to reduce the data volume.

In addition to that, you could start with 5 min data before the epilepsia, as suggested in the document, in the same dog (ex: dog 2). The first objective could be to detect which sensors are more significant in your case study, so that you can remove less representative ones (if they exist).

This is possible doing a correlation study between sensors. If a specific sensor doesn't have any correlation (=0 value) between its signals in the same dog, it would probably mean that it is not related to the epilepsia event.

Then, if you detect some correlations in specific sensors, you can start using multi variate models that could predict with more precision whether or not there will be epilepsia.

After doing good models on several significative sensors and in a few dogs, I suppose you can extend the predictions using 1 hour training.