1
$\begingroup$

I have a 15 timeseries datasets with 25-30 columns and is labeled by following a complex formula applied on the 25-30 columns.

When training, I split the datasets as training datasets and unseen datasets (12 datasets for training, 3 unseen datasets for inferencing)

only the data from the training datasets are used for training the lstm model. and the data from unseen datasets is used for inferencing after training.

After joining the training datasets, preprocessing and reshaping the data for lstm model, oversampling, and finally shuffling the data, I train-test-split the data 70% for training 30% for testing (70% of training datasets will be used for training, 30% of the training datasets will be used for testing).

After training the model I get 98% accuracy on training data as well as on testing data.

However, when I try to infer it on the unseen labeled dataset (not given to model while training and testing), it performs poorly

Things to note :-

  1. I am performing the same exact preprocessing and reshaping as I did for training.
  2. The data in the unseen dataset is somewhat similar to training data.
  3. The model is not overfitting, I looked at the training and validation loss graph, and it is nowhere near overfitting. If it was overfitting, i would not have gotten 98% accuracy on testing data.
  4. the training data is diverse enough. I synthetically generated the training data from the original data such that the values in columns are hitting all the ranges i.e. each column has normal distribution of data.

I want my LSTM model to learn the underlying formula rather than memorizing the data and its labels. If my model can learn the underlying formula, then no matter what data I give it, it will always follow the formula and always give correct output on unseen data.

this is my model:-

def create_lstmfcn_model(MaxTimeslice, H, LR, num_classes):

    ip = Input(shape=(MaxTimeslice, H))

    x = LSTM(64, return_sequences=True)(ip)
    x = LSTM(32, return_sequences=True)(x)
    
    x = LSTM(16)(x)

    y = Permute((2, 1))(ip)
    y = Conv1D(32, 20, padding='same')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(16, 15, padding='same')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(32, 10, padding='same')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = GlobalAveragePooling1D()(y)

    x = concatenate([x, y])
    x = Dense(units=16, activation='relu', )(x)
    

    multiclass_output = Dense(units=num_classes, activation='softmax',)(x)

    model = Model(inputs=ip, outputs=multiclass_output)

    model.compile(loss="categorical_crossentropy",
                  metrics=["accuracy"],
                  optimizer=RMSprop(learning_rate=LR))

    return model
$\endgroup$

1 Answer 1

3
$\begingroup$

Your model is likely overfitting and you get an inflated validation score of 98% because of a data leak. Your idea of setting aside 3 of the time series for testing is good, but you missed to do the same for the validation set where you do a random split of the training data. This will cause your validation data to be from time series your model have already seen training samples of.

I would suggest you do something like the following.

  • Split your 15 time series into 3 parts
  • Training set - Data from 10 full time series
  • Validation set - Data from 2 full time series
  • Test set - Data from 3 full time series

One thing to note is that this will not give you better performance, but will fix the issue where it looks like you are not overfitting.

Looking at your model code does not help in this case. The error is likely in how you split/pre-process/generate data, so showing that kind of code would help more.

$\endgroup$
2
  • $\begingroup$ Yes you are right, my validation set can be influenced by training data since I am first shuffling the data. I will keep separate datasets for my validation set. However, will doing so help my model predict better on unseen or test datasets? I read somewhere online that validation set will only help analyze the performance and not necessarily improve it. $\endgroup$ Feb 28 at 18:13
  • 2
    $\begingroup$ Yes, I added that info in an edit. It is correct that this will not increasing performance, but it will make you see the real issue, which is likely overfitting. So this is more of a step towards improving performance. $\endgroup$ Feb 28 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.