5
$\begingroup$

My task is to perform classify news articles as Interesting [1] or Uninteresting [0]. My training set has 4053 articles out of which 179 are Interesting. The validation set has 664 articles out of which 17 are Interesting. I have preprocessed the articles and converted to vectors using word2vec.

The CNN architecture is as follows:

sentence_length, vector_length = 500, 100
def create_convnet(img_path='../new_out/cnn_model_word2vec.png'):
    input_shape = Input(shape=(sentence_length, vector_length, 1))

    tower_1 = Conv2D(8, (vector_length, 3), padding='same', activation='relu')(input_shape)
    tower_1 = MaxPooling2D((1,vector_length-3+1), strides=(1, 1), padding='same')(tower_1)
    tower_1 = Dropout(0.25)(tower_1)

    tower_2 = Conv2D(8, (vector_length, 4), padding='same', activation='relu')(input_shape)
    tower_2 = MaxPooling2D((1,vector_length-4+1), strides=(1, 1), padding='same')(tower_2)
    tower_2 = Dropout(0.25)(tower_2)

    tower_3 = Conv2D(8, (vector_length, 5), padding='same', activation='relu')(input_shape)
    tower_3 = MaxPooling2D((1, vector_length-5+1), strides=(1, 1), padding='same')(tower_3)
    tower_3 = Dropout(0.25)(tower_3)

    merged = concatenate([tower_1, tower_2, tower_3], axis=1)
    merged = Flatten()(merged)
    dropout1 = Dropout(0.5)(merged)
    out = Dense(1, activation='sigmoid')(dropout1)

    model = Model(input_shape, out)
    plot_model(model, to_file=img_path)
    return model

some_model = create_convnet()
some_model.compile(loss=keras.losses.binary_crossentropy,
              optimizer='adam',
              metrics=['accuracy'])

The model predicts all articles in the validation set as Uninteresting [0]. The accuracy is 97.44% which is same as the ratio of Uninteresting articles in the validation set. I have tried variations of this architecture but still, the issue exists.

For experimentation, I predicted on the training data itself, for that too, it predicts all as Uninteresting [0]. Here are the logs for 10 epochs:

some_model.fit_generator(train_gen, train_steps, epochs=num_epoch, verbose=1, callbacks=callbacks_list, validation_data=val_gen, validation_steps=val_steps)
Epoch 1/10
254/253 [==============================] - 447s 2s/step - loss: 0.7119 - acc: 0.9555 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00001: val_loss improved from inf to 0.41266, saving model to ../new_out/cnn_model_word2vec
Epoch 2/10
254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00002: val_loss did not improve
Epoch 3/10
254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00003: val_loss did not improve

Epoch 00003: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
Epoch 4/10
254/253 [==============================] - 448s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00004: val_loss did not improve

Epoch 00004: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.
Epoch 5/10
254/253 [==============================] - 444s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00005: val_loss did not improve

Epoch 00005: ReduceLROnPlateau reducing learning rate to 1.0000000656873453e-06.
Epoch 6/10
254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00006: val_loss did not improve

Epoch 00006: ReduceLROnPlateau reducing learning rate to 1.0000001111620805e-07.
Epoch 7/10
254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00007: val_loss did not improve

Epoch 00007: ReduceLROnPlateau reducing learning rate to 1e-07.
Epoch 8/10
254/253 [==============================] - 443s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00008: val_loss did not improve

Epoch 00008: ReduceLROnPlateau reducing learning rate to 1e-07.
Epoch 9/10
254/253 [==============================] - 444s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00009: val_loss did not improve

Epoch 00009: ReduceLROnPlateau reducing learning rate to 1e-07.
Epoch 10/10
254/253 [==============================] - 440s 2s/step - loss: 0.7099 - acc: 0.9560 - val_loss: 0.4127 - val_acc: 0.9744

Epoch 00010: val_loss did not improve

Epoch 00010: ReduceLROnPlateau reducing learning rate to 1e-07.
Out[3]: <keras.callbacks.History at 0x7f19898b90f0>
$\endgroup$
  • $\begingroup$ That’s s lot of dropout going on. Try removing all the dropouts and only add them back if you are overfitting. $\endgroup$ – kbrose May 23 '18 at 13:04
6
$\begingroup$

Your dataset is highly imbalanced. Your optimization process is just minimizing the loss function, and cannot do better than a model that predicts uninteresting regardless of the input, due to the fact that your training set is very imbalanced. Moreover, you are not overfitting, since your training accuracy is lower than your validation accuracy.

In order to have a model that learns something less dummy than your model (and you might have to pay the price of having a lower accuracy), I would do the following: when providing a mini-batch to your optimizer, generate a mini-batch that is more balanced, that is, bias the elements you select towards the interesting articles. For instance, if your batch size is 64, ensure that it has 32 interesting elements and 32 uninteresting elements. Using this your network might start learning some features regarding the words in it, and in principle it should help you achieve a not so dummy predictor.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So, right now I have 179 interesting articles. Should I just increase the size of the training set by duplicating these 179 articles thus ensuring the batch will be more balanced? I duplicate by a factor of 20 to get the balance to around 50% $\endgroup$ – Shreyans Jasoriya Apr 27 '18 at 9:39
  • $\begingroup$ That's what I would do, but maybe you don't need it to be 50/50, as 40/60 or 30/70 might work as well. $\endgroup$ – David Masip Apr 27 '18 at 9:47
  • $\begingroup$ I augmented the data to be 40/60 split. Still, it predicts Uninteresting for all articles in validation set and even when training data is passed for prediction. $\endgroup$ – Shreyans Jasoriya Apr 27 '18 at 10:48
  • $\begingroup$ One thing I see in your example is that the learning rate is ridiculously small, this might have something to do with it. If your training data is 40/60, I guess that your training accuracy is around 60%. Am I right? $\endgroup$ – David Masip Apr 27 '18 at 12:00
  • $\begingroup$ Yes, around 57%. $\endgroup$ – Shreyans Jasoriya Apr 27 '18 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.