3
$\begingroup$

I have implemented the model proposed in this article which is a text classification model that uses sentence representation rather than only word representation to classify texts.

model=tf.keras.Sequential()
embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
model.add(TimeDistributed(embeding_layer))
model.add(TimeDistributed(tf.keras.layers.LSTM(50)))
model.add(tf.keras.layers.Bidirectional(costumized_lstm.Costumized_LSTM(50)))
model.add(layers.Dense(6,activation='softmax'))
opt=tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
self.model=model

and I use a dataset with 40000 documents with 6 different labels to train it. (30000 for train and 10000 for the test). I uses a pretrained word embeding and the input for this model is (sample,sentences,words). it achieves 84% accuracy. the problem is that I can achieve this accuracy very easily with this simple model:

        model=tf.keras.Sequential()
    embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
    model.add(embeding_layer)
    model.add(tf.keras.layers.Bidirectional(layers.LSTM(50)))
    model.add(layers.Dense(6,activation='softmax'))
    opt=tf.keras.optimizers.RMSprop(learning_rate=0.001)
    model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
    self.model=model

this one is not based on sentence representation and the input for this model is (sample, words). what is the first model ? is my implementation wrong? what should I do?

the training process for both models is as below picture. I also have used every trick to overcome overfitting but I haven't got any results. any suggestions please? enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ Hi, how do results look like for validation data? and does it change anything if you use the same optimizer, e.g. Adam, in both models? $\endgroup$
    – Jonathan
    Commented Dec 7, 2019 at 11:56
  • $\begingroup$ the result for validation data is shown in the above figure by the orange color, and there is not much different in the result by changing optimizer. $\endgroup$
    – jalil asadi
    Commented Dec 7, 2019 at 16:49

2 Answers 2

Reset to default
1
$\begingroup$

If you can achieve high results with a simpler model, it is great news! Always choose simpler models because they may be wrong on fewer things than complex ones (Occam's Razor).

Nothing to be worried about, this can happen.

Obviously, it is always possible that there is an implementation problem, so always make sure your code works better.

Reproducing results from papers is always difficult since many things can differ between your work and theirs:

  • Code might be different
  • Data might be different
  • Package versions may be different
  • Randomness might be controlled differently
  • Weights may be initialized differently
  • Data might be fed into the model differently
  • Hardware may be different which, especially with GPU can cause slight difference in results
Improve this answer
$\endgroup$
4
  • $\begingroup$ I understand, but the question is why the advanced model does not produce better results? it supposes to be better than the simpler model as is proved in the mentioned article. the advanced model not only captures the semantic between words, it captures the semantic between sentences too, so in theory, it should produce better results than a simpler version as is proved in the mentioned article, but why I can not achieve that? is my implementation wrong ? can you please check the article ? $\endgroup$
    – jalil asadi
    Commented Nov 29, 2019 at 16:00
  • $\begingroup$ I don't know why it doesn't produce better results. Making a model more complex does not always equate to better results on a specific problem. You use the word prove a lot but this is a strong word. The better shows that the model can produce good results. It doesn't prove anything at all. As for the differences with the paper, unless you use the exact same data, exact same initialization weights, and exact same random seed, your results may differ from their. $\endgroup$
    – Valentin Calomme
    Commented Nov 29, 2019 at 16:41
  • $\begingroup$ you are right, I am not a native English speaker so sorry for the poor choice of words, but I have chosen this paper as the basis for my thesis and now I am stuck. I do not know what to do. do you have any suggestions to improve the results? $\endgroup$
    – jalil asadi
    Commented Nov 29, 2019 at 16:49
  • $\begingroup$ You can check everything carefully, test with other languages perhaps, and if the simpler model is indeed better, it is an interesting result for the thesis. Also if you were able to implement this model, you should be able to learn enough Java to test the authors' solution. $\endgroup$
    – Valentas
    Commented Jan 10, 2020 at 17:46
0
$\begingroup$

Here are a couple of steps I suggest to take:

  1. Compare your data to the data used in the paper. Your dataset contains only 40k examples while theirs range from ~335k to ~1.57m. So it could be that your dataset is just too small for the more complex model. Looking at accuracy scores for training and validation data separately can help to figure this out too. Moreover, I'd check qualitative aspects, e.g. is your data actually rating/review-related? Or are you processing data which has very different semantics or a different structure?

  2. Test your data with the author's implementation. The resource link in their paper does not seem to work but I'd try to get their code and use their implementation to see if there is a relevant difference to yours and how it performs on your dataset.

  3. Test your implementation with the author's dataset. This way you can further validate the first bullet point, i.e. check to which degree the observed behavior is driven by differences in the datasets.

Improve this answer
$\endgroup$
8
  • $\begingroup$ thank you for your answer. I changed the dataset to 70k but still no change in the result, and both models produce the same results. my dataset is an official dataset in Persian, which is used for classification and has been used in a lot of papers. unfortunately, authors implementation is in Java language which I am not familiar with, I tried to figure it out but I couldn't. I even tried to use the attention layer on lstm model and hope to improve the result but I got the same result. ir.hit.edu.cn/~dytang/paper/emnlp2015/codes.zip this is the link of his codes in java. $\endgroup$
    – jalil asadi
    Commented Dec 7, 2019 at 17:05
  • $\begingroup$ I noticed that by increasing the size of the data set, the final accuracy of both models on the validation set reduces almost by 2%. more interestingly I noticed that using a random embedding layer produces better accuracy than using pre-trained word vectors as the embedding layer. is it normal ? $\endgroup$
    – jalil asadi
    Commented Dec 7, 2019 at 17:13
  • $\begingroup$ I increased the dataset to 200k and still no difference in the results for both models. they both achieve 74% accuracy on validation. even when I put an attention layer I get the same accuracy! $\endgroup$
    – jalil asadi
    Commented Dec 7, 2019 at 22:59
  • $\begingroup$ And is your dataset also related to reviews/ratings? And how do your accuracy graphs look like for training data? @jalilasadi $\endgroup$
    – Jonathan
    Commented Dec 8, 2019 at 21:41
  • $\begingroup$ yes they are relevant. there is a picture of the performance of second model above, the gray line is for training and the orange line is for test data. it perform very well on training but not on test. I also used every technic for reducing overfitting but I always get the same result. $\endgroup$
    – jalil asadi
    Commented Dec 8, 2019 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.