1
$\begingroup$

I see one paradox here. If we use Train/Test split and evaluate our Test data, we might get a good score, but any further prediction will not be credible, because model didn't train the Test data and include it's sequences in memory.

On the other side, we can train the data on Train and Test sequence as train data, but then we can not evaluate our predictions, because we have no testing reference.

How do you properly predict LSTM models?

$\endgroup$
  • $\begingroup$ Isn't this what K-fold cross validation is for? $\endgroup$ – Kari Jul 5 '18 at 16:25
2
$\begingroup$

You seek to augment the external validity of your model.

The most common way of doing so is by applying k-fold cross-validation to verify that your model generalizes well on unseen data.

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation.

This will reduce the variance of your model and will reduce its error on unseen data.

| improve this answer | |
$\endgroup$
0
$\begingroup$

This situation is common in a generic modeling setting, not only for LSTM.

In the development phase, the model is built using training data, and test data is used to estimate the model quality. Post this, the model is trained on the entire data, and this updated model is used for prediction on new data points.

| improve this answer | |
$\endgroup$
0
$\begingroup$

In general, effective learning is all about making the training error small and the gap between training and test error small.
By test data, we mean examples that your model has never seen before. so you need development (validation) set, to fine-tune your hyperparameters such hidden cells, the number of layers, learning rate, etc.
Split the training data into train/dev sets, be careful test set must always be generated from the same data distribution that generates your train/dev sets.
LSTM might overfit your dataset, start with vanilla RNN, or small GRU.
Use early stopping to stop training when the loss of the validation examples stop decreasing.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.