3
$\begingroup$

I'm using four years of data, training on the first 3 and testing on the fourth. Using LSTM w/ Keras. My test data set (which has no overlap at all with the training) is consistently performing better than my training data. How should I interpret this? It seems very unusual. Here's the trail end of the model output. You can see my training accuracy for a given epoch hovers around 80%, but test output jumps to about 86%:

Epoch 8/10
9092/9092 [==============================] - 9s 964us/step - loss: 0.9870 - acc: 0.8185
Epoch 9/10
9092/9092 [==============================] - 9s 1ms/step - loss: 0.9670 - acc: 0.7996
Epoch 10/10
9092/9092 [==============================] - 9s 937us/step - loss: 0.9799 - acc: 0.7895
Test Set Accuracy: 85.96%

predicted     0    1
actual              
0          2639  238
1           211  111

Edit: Here's my code to create & compile the model:

embedding_vector_length = 32
days = 30

model = Sequential()
model.add(Embedding(2080, embedding_vector_length, input_length=days)
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
model.fit(train_x, train_y, epochs=3, batch_size=64,class_weight={0:1.,1:1}) 
scores = model.evaluate(test_x, test_y, verbose=0)
print("Test Set Accuracy: %.2f%%" % (scores[1]*100))
$\endgroup$
2
$\begingroup$

I assume you're using structured data (numerical, categorical, nominal, ordinal..): - It's probably due to class imbalance. - If you use Scikit-Learn, you can add class_weight = "balanced" which will automatically weigh classes inversely proportional to their frequency. - Testing this should confirm if it's a class imbalance problem.

PS: Francois Chollet (create of Keras) states that traditional algorithms are superior to Deep Learning for structured data. Personally, with structured data, I've never been able to match the performance of XGBoost with Deep Learning.

enter image description here

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ My data is a time series, the problem is binary classification. Where would that put me with respect to deep learning & something like XGBoost? I've tried feature extraction (tsfresh) on the series data and used that with traditional ML, and not had very good results at all. Of course there might not be much signal to distinguish between the two classifications. $\endgroup$ – Jim the fourth May 24 '19 at 14:04
  • $\begingroup$ I don't get your question. If it's a classification problem, run a Random Forest with SK-Learn and set the class_weight to "balanced". Your test accuracy should be lower than your train accuracy. If that's the case, you know what's the issue with Deep Learning. $\endgroup$ – FrancoSwiss May 24 '19 at 19:53
  • 1
    $\begingroup$ Thanks, I'll give it a shot. I'm actually more comfortable with traditional algorithms, but not with time series data. It looked like LTSM might be a good candidate, but perhaps not. I appreciate your input, (and on my other question too) $\endgroup$ – Jim the fourth May 25 '19 at 4:28
  • $\begingroup$ Sure. Good luck! $\endgroup$ – FrancoSwiss May 25 '19 at 8:00
4
$\begingroup$

Your test-set is unlucky.

Use Cross-Validation

Also make sure to handle proper/random split for test and train data. So that it's not biased

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ Very likely this, +1. OP does not tell us how he generated the test set. That said, pytorch will already perform cross validation during .fit(). You do not want to do full k-fold over all data during the training, otherwise your hyperparameters can overfit to the datatset (instead of just parameters overfitting the dataset in a single model). OP should just manually check a handful of other splits, if this discrepancy in test and validation score happens again then one must search for different causes (e.g. a small number of outliers) $\endgroup$ – grochmal May 23 '19 at 16:59
  • $\begingroup$ Thanks, this is in fact the case, but my test set mimics the conditions under which I work. The most recent year's data shows more of the less-likely outcome than prior years. That is expected to continue, and the nature of the data is annual batches. So if my model can't use prior years to predict the current year, which is what my current train-test split mimics, then I'm not sure how useful it will be. $\endgroup$ – Jim the fourth May 25 '19 at 4:26
2
$\begingroup$

It seems your model is biased towards one class and some how on fourth year(testing data) you are getting more samples of biased class. You can also observe such problem due to deficiency of training data.

Probable solution: Add more samples in training data.

Note: You can also try model training after shuffle your all 4 years of data and split into training vs testing data.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ It is biased, but weighing in favor of the unlikely category actually makes the train-test accuracy issue worse. Shuffling all four years wouldn't accurately mimic the conditions I work under. I will need to predict year 5 all at once, when it arrives, using 1-4. So, I need to be able to predict year 4 using only 1-3 to properly reproduce those conditions. $\endgroup$ – Jim the fourth May 25 '19 at 4:34
1
$\begingroup$

You must be using some regularization techniques to avoid over-fitting of the training data. (For example : dropout regularization) It would be easy to analyze if you could post your code snippet.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks, I added the code that created & compiles the model. The loss function is binary_crossentropy. I guess I'm underfitting? But I would expect my test set accuracy to still be lower in that case. $\endgroup$ – Jim the fourth May 23 '19 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.