Changing model architecture doesn't impact results

Question

I am currently learning binary classification. The problem is classifying positive and negative movie reviews.

The dataset is 25,000 reviews with each review represented by 10,000 of the most used words. each review is transformed into multi-hot encoding and the labels are 1s and 0s for positive and negative reviews respectively.

This is the data preparation code:

from tensorflow.keras.datasets import imdb
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

import numpy as np
def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))
    for i, sequence in enumerate(sequences):
        for j in sequence:
            results[i, j] = 1
    return results

x_train = vectorize_sequences(train_data)
x_test = vectorize_sequences(test_data)

y_train = np.asarray(train_labels).astype("float32")
y_test = np.asarray(test_labels).astype("float32")

x_val = x_train[:10000]
partial_x_train = x_train[10000:]
y_val = y_train[:10000]
partial_y_train = y_train[10000:]

Now this is the initial model and evaluation:

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Dense(16, activation="relu"),
    layers.Dense(16, activation="relu"),
    layers.Dense(1, activation="sigmoid")
])

model.compile(optimizer="rmsprop",
              loss="binary_crossentropy",
              metrics=["accuracy"])

history = model.fit(partial_x_train,
                    partial_y_train,
                    epochs=20,
                    batch_size=512,
                    validation_data=(x_val,y_val))

results = model.evaluate(x_test,y_test)

With this model, if I only run 4 epochs I get the best loss and accuracy on the validation data and therefore on the test - accuracy 88%.

Now when I try different variations of the model architecture For example: different number of layers, number of units, using MSE loss instead of binary_crossentropy loss, using tanh instead of relu.

I don't get significant differences between all the models except sometimes different epochs are have the best loss and accuracy on the validation but the values of the best loss and accuracy doesn't change much.

Am I missing something or the different architectures really don't impact the predictions very much?

Answer 1

there are several aspects that impact the model results. The architecture is one of them, with a high influence. My guess is that your model is still too shallow (2 layers of 16 neurons each is very small).

Typically, if you see no significant change in the results when changing a basic element in the architecture (for example, number of layers) it's because the architectures that you tried have very similar complexity (in terms of number of parameters).

Try with 100 neurons per layer, then with 500 neurons per layer, and then see how the performance changes when increasing the number of layers.

You might want to increase also the number of epochs. 20 is also very low. Try for 100 with early stopping on the validation loss.

-- For the rest:

• If its a classification problem, you should definitely stick to cross entropy loss, and not MSE (MSE is used mostly for regression problems).
• Check also your initial learning rate. If its too big, the model will just keep jumping between bad performances. Try initial learning rate of 1e-3. Also, try adding a learning rate scheduler (decreases the learning rate with the number of epochs).

---

If you still face the problem, there might be a problem with how you process your data.