0
$\begingroup$

After going through the tutorials of tensorflow I decided to do my own project. For this I did the following:

  • I have scraped movie reviews from a polish site called filmweb. I got a dataset o 5400 movie reviews with a rating in range 1-10
  • I used the “Morfeusz” python bindings to tokenize and lemmatize the text. I have stripped most basic stop words.
  • I mapped most used 10000 lemmas to numbers and put the data in lists.
  • Randomly split the data into two equal sets.
  • My training data is the movie review lemmas padded with zeroes (like in the tutorial). The expected result is the rating.
  • My model is
    vocab_size = 10000

    model = keras.Sequential()
    model.add(keras.layers.Embedding(vocab_size, 16))
    model.add(keras.layers.GlobalAveragePooling1D())
    model.add(
        keras.layers.Dense(16, activation=tf.nn.relu)
    )
    model.add(keras.layers.Dense(1))

    optimizer = tf.keras.optimizers.RMSprop(0.001)


    model.compile(optimizer=optimizer, 
              loss='mean_squared_error',
              metrics=['accuracy'])

After running the optimization process i see the loss metric dropping quite fast. However the accuracy is well below 0.1 and after I check the predictions of this network, they are useless. The model predicts all the ratings to be in range of 6-7. So I understand that the network “optimized” by simply always guessing the mean value.

I wonder what should be my next step to improve. Is my model wrong for this task? Should I somehow normalize or augment my data? Or is the dataset size simply too small to get any meaningful result?

EDIT: this is the link to my dataset and the script that generated it: https://www.dropbox.com/s/csyo934hhtbxxu9/fw-data.tar.bz2?dl=0

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Could you post atleast a small screensot of your dataset? $\endgroup$
    – Bhakti
    Commented Jun 13, 2019 at 9:58

1 Answer 1

Reset to default
1
$\begingroup$

What you are doing right now is regression on the review score, therefore accuracy is not a suitable metric to assess performance. Your model is optimizing the mean squared error, thus it makes sense that it is predicting the (almost) average review score.

Since you are interested in the accuracy on the rating's prediction, you should set your output layer as a softmax layer with 10 possible outputs, as in:

model.add(keras.layers.Dense(10), activation='softmax')

Then you can optimize the categorical_crossentropy of your model, and report the accuracy. In this way, you'll do classification of the ratings.

model.compile(optimizer=optimizer, 
              loss='categorical_crossentropy',
              metrics=['accuracy'])
Improve this answer
$\endgroup$
5
  • $\begingroup$ This causes the accuracy to be about 0.2 And as before – all the predictions are almost the same: ``` [[0.01122459 0.02773546 0.06952148 0.09368683 0.12801726 0.19622442 0.21538338 0.15694933 0.0623847 0.0388726 ] [0.01395655 0.03201564 0.06793261 0.09513801 0.12118061 0.17481872 0.21145028 0.17532745 0.0666419 0.04153823] [0.01075333 0.02595716 0.06822971 0.0892809 0.13129255 0.19443946 0.22296897 0.15693714 0.06288221 0.03725852] [0.00932392 0.02411646 0.06628047 0.09013111 0.13089287 0.20698686 0.2239832 0.1541867 0.0588787 0.03521968] ``` $\endgroup$
    – zefciu
    Commented Jun 13, 2019 at 13:14
  • $\begingroup$ What is the frequency of labels in your data? If this is really skewed this might be the problem. Another think you can try is to make the problem easier, i.e. try to predict 4 classes -- for example, rating 1-3, 4-5, 6-7, 8-10 (very bad, bad, ok, good) and see if this gives good result. In that case, you just might need more data for extending to the 10 classes. $\endgroup$
    – rob_med
    Commented Jun 13, 2019 at 13:18
  • $\begingroup$ Yes, the rating frequency is pretty skewed: [40, 127, 358, 440, 763, 1176, 1283, 869, 298, 180]. I already tried to categorise into 3 classes. Should I try to get classes with similar size? Or is there any other way to normalize my data? $\endgroup$
    – zefciu
    Commented Jun 13, 2019 at 13:41
  • 1
    $\begingroup$ So the network is predicting the most frequent classes, that is 6-7, given the skewness. It also looks like your data is ~6000 samples, and you are using 10000 features. If this is the case, you're most likely also overfitting on the training data, since it is too wide (the model has way to many parameters w.r.t. the number of samples). You can try either reducing the number of lemmas you use, or gathering more data (but you'd need quite some more). Personally, I would start with a small set (say 50 reviews x rating) and ~100 lemmas and see how that goes. $\endgroup$
    – rob_med
    Commented Jun 13, 2019 at 14:03
  • $\begingroup$ With such a minuscule dataset I get accuracy ~0.2 and predictions that are almost all 0.25 $\endgroup$
    – zefciu
    Commented Jun 13, 2019 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.