0
$\begingroup$

I want to divide my time series dataset into training and test sets. The data is seasonal and very noisy. When I randomly split, the test and train samples do not resemble in their distributions. Sometimes, train data get most of the noisy/peak points and sometimes these points go to test data. Can I split the data into train/test by calculating how much information the two sets hold. The information can be quantified by e.g. Shannon entropy or standard deviation etc. When I tried to search literature around this, I could not find any reference?

$\endgroup$
1
  • $\begingroup$ If you are dealing with time series dataset, would it be nature that you split the data using a certain point on the time line (before which will be the training data, and after which will be the validation/test data)? If there are seasonal trends, you may want to make sure that your training data covers a full cycle. $\endgroup$
    – Albert
    Jul 15, 2021 at 11:19

2 Answers 2

0
$\begingroup$

In theory, the very first thing to do should be fixing noises and seasonality. There are various ways of approaching the noises. If possible, try to understand the reason for noisy samples and extract the noise part from them manually. Or, use the smoothing algorithms to minimize the noises. In addition, you can create a dummy predictor variable that will be accountable for the noise part in the data. In time-series data, you can manually put 1 for the dates that you consider as noise and 0 for normal data points. You can read more about dummy variables here. However, it might be unhealthy to use dummy variables for improper cases.

To overcome seasonality, there are some powerful approaches, for example, X-13ARIMA-SEATS is a good method of extracting the seasonality from the time series.

If you want to do a split on the strength of some metric, you might encounter different possible scenarios. For example, if you would use some metric to split the data like variance or information then your test data would be very dissimilar compared to your train data. Thus, the test set would perform poorly since its characteristic is not like the train data. In other words, you might not objectively assess the performance of the model.

Lastly, as Albert mentioned in the comments, if you do a time-series prediction, then it would be better to split the data using the time. In other words, n consecutive samples would be your train data and m consecutive samples would be your test data.

$\endgroup$
1
  • $\begingroup$ Regarding your 3rd para: The way I am suggesting is not by maximizing the information in training set but by reducing the absolute difference between information that is contained in the training and test sets. For example if I split the whole data such that the difference in entropy of training and test is minimum. Does that make sense? Is there any literature around that? I understand the point you are making regarding seasonality/preprocessing etc. $\endgroup$ Jul 15, 2021 at 12:49
0
$\begingroup$

Very few of these highly noisy series might indicate they are

  1. unwanted outliers; or
  2. an anomaly, such as network intrusion in a dataset of network traffic.

In case 1 reason why they are present, and use outlier analysis techniques. In case 2 you should in fact make sure they are represented both in the train as well as the test set.

Validate you do not have a sampling bias in your dataset. Treating these samples differently might also lead to data snooping. Read more here.

$\endgroup$
1
  • $\begingroup$ Thanks for your answer. While you've provided some advice on how to deal with outliers, this doesn't answer the OP's question, which is about train/test splits and how to ensure the noise/information is evenly spread across the train and test sets. Can you update your answer to address these points? (From review). $\endgroup$
    – Lynn
    May 21, 2023 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.