0
$\begingroup$

I built a Keras model to predict hoax news and true news using the CNN-BERT Text Classification algorithm with Categorical Classification, with label 1 indicating a hoax and 0 indicating true news.

Although the model I created appears to have good training and validation accuracy results, when I make predictions on the test set, it does not appear to be able to predict.

Pretrained BERT model that i used

tfhub_handle_encoder = hub.load('https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-2_H-512_A-8/1')

tfhub_handle_preprocess = hub.load('https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')

Model

from tensorflow.keras import regularizers


def build_CNN_classifier_model():
    text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
    preprocessing_layer = hub.KerasLayer(tfhub_handle_preprocess, name='preprocessing')
    encoder_inputs = preprocessing_layer(text_input)
    encoder = hub.KerasLayer(tfhub_handle_encoder, trainable=True, name='BERT_encoder')
    outputs = encoder(encoder_inputs)
    net = outputs['pooled_output'] # [batch_size, 768].
    net = sequence_output = outputs["sequence_output"] #[batch_size, seq_length, 768]
      
    
    net = tf.keras.layers.Conv1D(32, (2), activation='relu')(net)
    net = tf.keras.layers.MaxPooling1D(2)(net)
#     net = tf.keras.layers.Dropout(0.1)(net)
    
    net = tf.keras.layers.Conv1D(64, (2), activation='relu')(net)
    net = tf.keras.layers.MaxPooling1D(2)(net)
    net = tf.keras.layers.Dropout(0.2)(net)
    
    net = tf.keras.layers.Conv1D(128, (2), activation='relu')(net)
    net = tf.keras.layers.MaxPooling1D(2)(net)
    net = tf.keras.layers.Dropout(0.2)(net)
    
#     net = tf.keras.layers.GlobalMaxPool1D()(net)
    
    net = tf.keras.layers.Flatten()(net)
    
    net = tf.keras.layers.Dense(256, activation="relu")(net)
                                #,kernel_regularizer=regularizers.L1L2(l1=1e-5, l2=1e-4),
                                #bias_regularizer=regularizers.L2(1e-4),
                                #activity_regularizer=regularizers.L2(1e-5))
                                
    net = tf.keras.layers.Dropout(0.1)(net)
    
    net = tf.keras.layers.Dense(128, activation="relu")(net)
                                #,kernel_regularizer=regularizers.L1L2(l1=1e-5, l2=1e-4),
                                #bias_regularizer=regularizers.L2(1e-4),
                                #activity_regularizer=regularizers.L2(1e-5))
                                
    net = tf.keras.layers.Dropout(0.1)(net)
    
#     net = tf.keras.layers.Dense(1, activation="sigmoid", name='classifier')(net)
    net = tf.keras.layers.Dense(2, activation="softmax", name='classifier')(net)
    
    return tf.keras.Model(text_input, net)

compile

from official.nlp import optimization  # to create AdamW optmizer

epochs = 10
steps_per_epoch = tf.data.experimental.cardinality(train_ds).numpy()
num_train_steps = steps_per_epoch * epochs
num_warmup_steps = int(0.1*num_train_steps)

init_lr = 1e-5
optimizer = optimization.create_optimizer(init_lr=init_lr,
                                          num_train_steps=num_train_steps,
                                          num_warmup_steps=num_warmup_steps,
                                          optimizer_type='adamw')

cnn_classifier_model.compile(optimizer=optimizer,
                             loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                             metrics=tf.keras.metrics.SparseCategoricalAccuracy('accuracy'))



print(f'Training model with {tfhub_handle_encoder}')
cnn_history = cnn_classifier_model.fit(x=train_ds,
                                       validation_data=val_ds,
                                       epochs=epochs,
                                       class_weight=class_weight
                                      )

Results

enter image description here

enter image description here

enter image description here

enter image description here

Test Prediction

enter image description here

it seems like my test model can't tell which one is hoax news and which one is real news, what's the problem?

Improve this question
$\endgroup$
3
  • $\begingroup$ You only tried 2 test samples? $\endgroup$
    – liakoyras
    Nov 16, 2022 at 15:50
  • $\begingroup$ I tried with different samples by replacing the text in the real and hoax datasets when predicting the dataset, the sample above is an example. $\endgroup$
    – AccelUp
    Nov 16, 2022 at 17:00
  • $\begingroup$ Try it with a thousand (or a million, I don't know how big your dataset is) samples and check your stats. If your training accuracy is 90%, the test should not be much lower (but is expected to be lower). But I don't think you can safely extract any conclusions for checking some samples manually. Maybe those specific examples where too hard. If everything is classified wrongly, then something is indeed wrong, but the information provided here does not suffice. $\endgroup$
    – liakoyras
    Nov 16, 2022 at 17:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.