0
$\begingroup$

I have this dataset that I'm trying to train a neural network on.

The problem is that since weekend dates are not available, I am not confident in whether the model is able to account for that. Moreover, if say the training set ends on a Friday and the horizon is set to 5, the model will predict for Sat-Wed which does not make sense (as weekends are irrelevant here). This is also causing a mismatch between the dates in my prediction and test dataframes:

                  ds  Autoformer
unique_id                       
1         2020-03-21    0.002096
1         2020-03-22    0.002087
1         2020-03-23    0.002164
1         2020-03-24    0.001982
1         2020-03-25    0.002581
             ds         y  unique_id
5072 2020-03-23  0.002860          1
5073 2020-03-24  0.000924          1
5074 2020-03-25  0.001324          1
5075 2020-03-26  0.001093          1
5076 2020-03-27  0.000570          1

I do not want to use interpolation as there are only 5k data points. The dataset will become really polluted on adding 2 interpolated points for every 5. Please suggest something.

I am using the Nixtla package to run these models in Python. Also, please help me understand what is the use of splitting training and test sets if the horizon is set. Like if the horizon is 5 days, what's the use of having 20-30% of your data as a test set? The way I see it you just need the last 5 observations.

$\endgroup$

1 Answer 1

2
$\begingroup$

I am not sure how you could define such a predictive model using Nixtla (I am unfamiliar with this software). However, tensorflow has a way you can mask specific timestamps. Perhaps this is what you're looking for. In your case, you would pad the sequences with some default dates or values and then tell your model in the masking layer that you want to ignore those sequences that were filled with your mask value.

Something in the ballpark of

def build_model():

model = Sequential()
model.add(Input(shape=(train_data_with_features_removed.shape[1], train_data_with_features_removed.shape[2])))
model.add(Masking(mask_value=-999))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.