0
$\begingroup$

I've been using Keras to do some timeseries predictive neural nets. One thing that's had me staggered is how I was getting an almost 99% accuracy rate on very very noisy and uncorrelated data (I was looking for accuracy of around 50% to be good!)...

What I've now found, is that after I trained my model on the training set, I was casually running model.evaluate(x_test, y_test) before then running a manual backtest on the data. The manual backtest on x_test and comparing it to the error in y_test then showed almost perfect accuracy...

Am I then correct in assuming that Keras when using the evaluate() function in Keras the original model is actually updated with the testing data as well? This seems a bit strange and counterintuitive to me?

$\endgroup$
1
  • $\begingroup$ Could you provide minimal working example? 99% accuracy on small training dataset suggests overfitting. How big is your dataset? Additionally evaluate does not have any reported side effects, so it should not update your model (that would be bad design anyway if it did). $\endgroup$ – Dmitrii I. Sep 19 '16 at 13:28
1
$\begingroup$

Keras doesn't update your model with testing data. It might be that your labels have been provided wrong in the test data, check the model.predict_classes() to get the output of your classes and crosscheck them with your actual output manually by picking random subset.

$\endgroup$
1
$\begingroup$

There is a difference between accuracy and validation accuracy. Add in a term "validation_split = .2" into the model.fit statement and see if the validation accuracy is still 99%. If you are dealing with truly uncorrelated data and are getting 99% accuracy then you are over fitting the dataset and need more Ys in your sample as compared to the number of X variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.