I see one paradox here. If we use Train/Test split and evaluate our Test data, we might get a good score, but any further prediction will not be credible, because model didn't train the Test data and include it's sequences in memory.
On the other side, we can train the data on Train and Test sequence as train data, but then we can not evaluate our predictions, because we have no testing reference.
How do you properly predict LSTM models?
You seek to augment the external validity of your model.
The most common way of doing so is by applying k-fold cross-validation to verify that your model generalizes well on unseen data.
In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation.
This will reduce the variance of your model and will reduce its error on unseen data.
This situation is common in a generic modeling setting, not only for LSTM.
In the development phase, the model is built using training data, and test data is used to estimate the model quality. Post this, the model is trained on the entire data, and this updated model is used for prediction on new data points.
How you break the "paradox" is as follows.
Independently on whether you use a hold-out test set or run Cross-Validation, keep in mind that those protocols are just meant to assess the performance of the model. However, after that assessment is done, you re-train your model on all the data. The implicit assumption is that the re-trained model will be at least as performing as the one trained on the reduced training set.
Therefore, in practice, this is a two-stage procedure and the issue that puzzles you does not persist.
"Test" is often misleadingly used instead of "validation". Even academic literature sometimes use them interchangeably, while they are not.
It is correct, as you say, to discard the validity of "test" data if we use it during training to decide when to stop the training.
In such case, the correct word is "validation" data. Validation data is by definition the data you use to verify your model can still generalize well. A model overfitting to its training data will have its validation loss increase.
However, to hold more reliable assumptions about generalization, we may use a held-out, so called, "test" set on which we test the model, once we decided to freeze the model's parameters for good.
But after doing so, you may wonder: why choosing one train/validation/test configuration rather than another random one from your dataset?
To answer this problem and get even stronger assumptions on the generalization capabilities of our model, we can use cross-validation which is considered a fairly robust evaluation method in machine learning.
Eventually, all assumptions we make about the loss of our model on some set of data is subject to statistics, in which generally we cannot draw clear conclusions.
In general, effective learning is all about making the training error small and the gap between training and test error small.
By test data, we mean examples that your model has never seen before. so you need development (validation) set, to fine-tune your hyperparameters such hidden cells, the number of layers, learning rate, etc.
Split the training data into train/dev sets, be careful test set must always be generated from the same data distribution that generates your train/dev sets.
LSTM might overfit your dataset, start with vanilla RNN, or small GRU.
Use early stopping to stop training when the loss of the validation examples stop decreasing.
