2
$\begingroup$

I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests and some very streamlined "analytic" tests pass, my code is not a mess, and my model is actually trying to do something meaningful with this data. I know that I cannot create a very good "representative" small subset which can totally mimick the original, but I want to make sure I find many of my model's base flaws with this data before training it on that huge dataset. Maybe having multiple different sized batches for different scopes of tests is an option too, I don't have any preferences.

What is the best strategy to create this subset? I think for a data that is not sequential, unlike mine, random downsampling of the datapoints might be a good thing, but I don't know what is a good practice in time Series data. Should I just choose a small frame of time as the new dataset? What about casuality? How to sample according to class imbalance? These are the first questions that come to my mind. But feel free to expand on even more questions.

Edit: What I am working on is this dataset. The dataset is quite large, and I want to effectively choose a subset from it. The task is to detect seizures. One option is to the number of subjects, I think. But I am open to all options that you might suggest!

$\endgroup$

1 Answer 1

1
$\begingroup$

In your case, you have first to deal with the biological data complexity.

I don't know the minimum sampling rate to detect brain epilepsia or any brain behavior. I would recommend to study some articles to know the best practices about EEG signal analysis like this one : https://www.frontiersin.org/articles/10.3389/fneur.2020.00375/full

Maybe there are good practices to reduce the data volume.

In addition to that, you could start with 5 min data before the epilepsia, as suggested in the document, in the same dog (ex: dog 2). The first objective could be to detect which sensors are more significant in your case study, so that you can remove less representative ones (if they exist).

This is possible doing a correlation study between sensors. If a specific sensor doesn't have any correlation (=0 value) between its signals in the same dog, it would probably mean that it is not related to the epilepsia event.

Then, if you detect some correlations in specific sensors, you can start using multi variate models that could predict with more precision whether or not there will be epilepsia.

After doing good models on several significative sensors and in a few dogs, I suppose you can extend the predictions using 1 hour training.

$\endgroup$
4
  • $\begingroup$ I'm trying to create a classifier on a multi-channel biomedical signal, the original dataset is big, but I'm planning to create a subset that is good enough to give me a very first-hand experience of what my model is doing, and then train it on the original dataset and do the final evaluations. Is downsampling enough in my case? $\endgroup$
    – Farhood ET
    Sep 14, 2021 at 10:25
  • 1
    $\begingroup$ If your data set is time dependent, you can't take a random sample over the whole data set, because it wouldn't detect time series'dynamics using random time records. But you can reduce the whole volume by taking mean values over specific range of time. I would need an example of your data to help you further. You can edit the original post. $\endgroup$ Sep 14, 2021 at 12:45
  • $\begingroup$ I've updated my post. Thanks! $\endgroup$
    – Farhood ET
    Sep 14, 2021 at 13:11
  • 1
    $\begingroup$ Thanks, I've updated my answer accordingly. $\endgroup$ Sep 14, 2021 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.