Why my validation loss and accuracy decays over epochs?

Question

Im trying to build 2 simple networks with cleaned dataset for tweets sentiment classification(0/1):

• one with all dense layers(binary bag of words)
• another with RNN layer(embedding layer). But it both cases the validation loss and accuracy are always low. Pasting the code and graphs for reference.

    keras.layers.Dense(250, input_shape=(500,),activation='relu'),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(250, activation='relu'),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(1, activation='sigmoid')
])

model2 = keras.models.Sequential([
            keras.layers.Embedding(input_dim=vocabulary,
                     output_dim=EMBEDDING_DIM,
                     input_length=max_length,
                     mask_zero=True),
            keras.layers.SimpleRNN(500, activation='relu'),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(100, activation='relu'),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(100, activation='relu'),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(1, activation='sigmoid')           
])
[![enter image description here][1]][1]```

Tried increasing network complexity by adding more layers and added dropouts. Still nothing increase the loss and accuracy. What am i missing?


  [1]: https://i.stack.imgur.com/tKlYS.png

Answer 1

You are experiencing a lot of overfitting on the training set here.

I would go back and see if there are any inherent issues with the data (scaling? class imbalance? etc.) before diving into modeling.

Also, be aware that if you are using built-in validation (e.g. validation_split=0.3) in TF/Keras, it does not shuffle the data or randomly sample and so if your data are sorted by class label this can seriously impact your results.