Original blog post claims that it is possible to get 95% accuracy on the validation set (20 Newsgroup dataset) after only 2 epochs using pretrained word embeddings (glove.6B.100d). All code is located here. I did not changed anything in this example but getting only 40% accuracy on the validation set after 2 epochs and 75% after 10 epochs. I can't get to 95% accuracy even after 20 epochs. Switching from tensorflow to theano backend do not make any significant change. I'm using Keras 2.0.2, tensorflow-gpu 1.0.1, theano 0.9.0, python 2.7.12. What I'm doing wrong?
9
-
$\begingroup$ soemthing similar happens to me while trying another classifier. I restarted the kermel and all was good (strange but true). $\endgroup$– user1043144Commented Mar 27, 2017 at 19:25
-
$\begingroup$ switched from notebook to desktop PC with another CPU, GPU and tensorflow-gpu 1.1.0rc0, still getting same results. $\endgroup$– enclisCommented Mar 29, 2017 at 16:32
-
$\begingroup$ I tried the script myself and get the same results as you (acc: 0.81 and val_acc: 0.72) and again (acc: 0.81 and val_acc: 0.7087). apparently others can replicate it github.com/kimardenmiller/NLP_CNN/blob/master/Embeddings/… and here groups.google.com/forum/#!msg/keras-users/s_veHQbyQmc/… I will take a closer look $\endgroup$– user1043144Commented Mar 30, 2017 at 16:07
-
$\begingroup$ Both links correspond to Keras version before 2.0. Maybe we should try to roll back to earlier versions? $\endgroup$– enclisCommented Mar 30, 2017 at 17:59
-
$\begingroup$ I found a small difference but had not time to check it . embedding_matrix = np.zeros((num_words + 1 , EMBEDDING_DIM)). Take a look and let me know $\endgroup$– user1043144Commented Mar 30, 2017 at 18:07
|
Show 4 more comments
2 Answers
$\begingroup$
$\endgroup$
The code has been changed to remove headers. See comment on github:
"Newsgroups message contains header like 'Newsgroups: alt.atheism', which inflates the accuracy to 0.95 (2 epochs). After removing the header, the val accuracy is 0.47 (2 epochs) and 0.71 (10 epochs)."
https://github.com/fchollet/keras/pull/5585
This confused me for days!
$\begingroup$
$\endgroup$
It may be the version of the 20 newsgroups data used. There are two versions of the dataset available from http://qwone.com/~jason/20Newsgroups/: (1) the original (2) another with duplicates removed and headers removed that give away the actual group. This dataset is also already separated into train and test.
When I load the original data set (with duplicates and headers) and use the code you reference I can get to ~93-95% after three epochs.
When I load the second version of the dataset (training only), it takes many more epochs to converge, with a validation accuracy of only about 55% after 10 epochs. Validation accuracy was still on an upward trajectory, so you should be able to improve on these results.
As such, I think that that the duplicates/filtering are significantly skewing the results from the blog you mentioned. The classification is easier when the header contains which news group in which it should be classified. The lesser accuracy is a more realistic evaluation of the data.
Also note that if you load the dataset using scikit-learn (sklearn.datasets.fetch_20newsgroups), it loads the second version of the dataset. A tf-idf model with multinomial Naive Bayes gets to about 50% on the data, so the RNN is still out performing a baseline model. Though, 95% seems not realistic on the second version of 20 newsgroups.
