1
$\begingroup$

Problem Summary

My model is built and compiled properly but gets the NaN validation loss on all epochs. The training set accuracy is also infinitesimally small and keeps decreasing. I couldn't find a mistake in the tokenization, embedding, and model-building code. I am using the training set of BBC articles data set from Kaggle: BBC News Classification. I only use the training data file.

  • Some common words like a, as, are, at, and be are removed after loading the file into a variable.
  • I was able to get a proper article out of the tokenized training data using the text vectorizer vocabulary.
  • Manually reviewing the first sentence and labels, I found they match the file I stored in my Google Drive.
  • The model outputs one array of five decimals per input using a soft-max activation. There are five output neurons, one for each article class. I used the SparseCategoricalCrossEntropy loss function.
  • The labels are vectorized using a separate TextVectorization() instance. Here, I adapt the vectorizer to ALL labels and tokenize both the training and validation labels.

Text Vectorization Check

I turn the articles into sequences of integers based on a vocabulary using TextVectorization(). I adapt the vectorizer to the training split only and then use it to tokenize the training and testing articles.

# NUM_WORDS is the maximum number of words allowed in the vocabulary.
NUM_WORDS = 1024 # I tried values from 1024 to 65536. No effect.
EMBEDDING_DIM = 64
# MAXLEN is the maximum length of the input sequences.
MAXLEN = 256
PADDING = 'post'
OOV_TOKEN = "[UNK]"
TRAINING_SPLIT = .8

def fit_tokenizer(train_sentences, num_words, max_len):

    tokenizer = tf.keras.layers.TextVectorization(
        max_tokens=num_words,
        standardize='lower_and_strip_punctuation',
        split='whitespace',
        output_mode='int',
        output_sequence_length=max_len,
        pad_to_max_tokens=True
    )

    # Fit the tokenizer to the training sentences
    tokenizer.adapt(train_sentences)

    return tokenizer
  • After calling the adapt() method of the TextVectorization() layer on train_sentences, I use TextVectorization.call():
tokenizer = fit_tokenizer(train_sentences, NUM_WORDS, MAXLEN)
train_padded_seq = tokenizer.call(train_sentences)
val_padded_seq = tokenizer.call(val_sentences)

The output shape looks fine: (1192, 256) and (298, 256). Each article is padded or truncated to be 256 integers long. The training and evaluation data are tensors. train_padded_seq.shape outputs TensorShape([1192, 256]). print(train_padded_seq[0]) outputs a sequence of integers with shape=(256,), dtype=int64.

  • When tokenizing the labels, I tried subtracting one from every label to make sure they all started at 0, but that made the accuracy exponentially low on the first epoch instead of a low decimal.

def tokenize_labels(all_labels, split_labels):
    label_tokenizer = tf.keras.layers.TextVectorization(
        max_tokens=NUM_WORDS,
        standardize='lower_and_strip_punctuation',
        split='whitespace',
        output_mode='int',
        output_sequence_length=1,
        pad_to_max_tokens=False
    )
    
    label_tokenizer.adapt(all_labels)
    label_seq_np = label_tokenizer.call(split_labels)
    # label_seq_np = np.array([int_label - 1 for int_label in label_seq], dtype=np.int8)

    return label_seq_np

I tokenize based on the full list of labels.

train_label_seq = tokenize_labels(labels, train_labels)
val_label_seq = tokenize_labels(labels, val_labels)

Model Building

  • The model compiles and runs. Its accuracy was terrible when I used the wrong metric: accuracy. It should be "categorical_accuracy." With accuracy, the best performance is during the first epoch when I get

loss: nan - accuracy: 0.0076 - val_loss: nan - val_accuracy: 0.0000e+00.

When I use categorical accuracy as my metric, the losses for the training and validation sets immediately go to NaN and both sets have an accuracy of one:

loss: nan - categorical_accuracy: 1.0000 - val_loss: nan - val_categorical_accuracy: 1.0000.

In both cases, running model.predict() gives me nothing but nans in the output.

model = create_model(NUM_WORDS, EMBEDDING_DIM, MAXLEN)
history = model.fit(train_padded_seq, train_label_seq, epochs=3, validation_data=(val_padded_seq, val_label_seq))

model.predict(train_padded_seq)[0].shape gives (5,).

model.predict(train_padded_seq)[0, : 5] gives array([nan, nan, nan, nan, nan], dtype=float32).

Changing the maximum sequence length, embedding dimensions, and vocabulary size (NUM_WORDS) did not change the results much. The results were always nan on a prediction and accuracy never went above very small decimals.


def create_model(num_words, embedding_dim, maxlen):
    tf.random.set_seed(123)

    embedding_layer = tf.keras.layers.Embedding(
        input_dim=num_words,
        output_dim=embedding_dim,
        input_length=maxlen,
        mask_zero=True,
        name='embedding',
    )

    # input_sin_pos_encoding_layer = keras_nlp.layers.SinePositionEncoding()

    sigmoid_layer = tf.keras.layers.Dense(
        units = 5,
        activation=tf.keras.activations.softmax,
        use_bias=True,
        kernel_initializer='glorot_uniform',
        bias_initializer='zeros',
        name='1_vs_5_sigmoid'
    )

    model = tf.keras.Sequential([
        tf.keras.layers.Input(shape=(maxlen)),
        # tf.keras.layers.InputLayer(input_shape=(maxlen)),
        embedding_layer,
        tf.keras.layers.Bidirectional(
            layer=tf.keras.layers.LSTM(64, return_sequences=False)
        ),
        tf.keras.layers.Flatten(name='flatten'),
        tf.keras.layers.Dense(
            units=64,
            activation=tf.keras.activations.relu,
            name='Dense_1_64'),
        tf.keras.layers.Dense(
            units=16,
            activation=tf.keras.activations.relu,
            name='Dense_2_16'),
        sigmoid_layer,
    ], name='model')

    optimizer_adam=tf.keras.optimizers.Adam(
        learning_rate=0.001,
        beta_1=0.9,
        beta_2=0.999,
        epsilon=1e-07,
        amsgrad=False,
        name='Adam'
    )

    model.compile(
        loss=tf.keras.losses.CategoricalCrossentropy(),
        # loss=tf.keras.losses.SparseCategoricalCrossentropy(),
        optimizer=optimizer_adam,
        # metrics=[tf.keras.metrics.CategoricalAccuracy()]
        metrics=['sparse_categorical_accuracy']
    )

    print('Summary:\n', model.summary())
    return model
$\endgroup$

1 Answer 1

0
$\begingroup$

The reason accuracy kept decreasing after every epoch was that the metrics function and loss function were not set to SparseCategoricalCrossentropy(). The output may have been NaN because the probability of a text matching a certain label was always extremely low. Using CategoricalCrossEntropy() as the loss function resulted in an error saying "Shapes (None, 1) and (None, 5) are incompatible."

The proper metric for this problem is SparseCategoricalCrossentropy() since the output of the final layer is a set of probabilities while the true label is just one integer. Using other metrics may also have caused NaN or very low accuracy. Using the Categorical Accuracy metric resulted in training accuracy below one percent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.