I'm doing a classification of time sequenced sensor data (in Python), where I'm segmenting the sensordata into samples, with a certain window-length (e.g. 3 seconds). However, the samples are also overlapping each other. For example, the first sample is 0s $\rightarrow$ 3s, second is 2.7s $\rightarrow$ 5.7s...
I'm wondering now, how I can do a proper train test split for this samples. Right now, I see two ways to do this:
Split the samples without shuffling them first and drop the sample "at the border", to avoid overlap between train and test data. However, this is not optimal because I would like to have samples from all over the dataset for both the training and the test set. Otherwise I would just test the classifier on a sequence of the data, that might be very different from the sequence used for training.
Shuffle the samples first and then split them into training in test. This would result in an overlap between training and test data and would therefore produce overly optimistic results.
Does anyone have another idea how this could be done?