Improve the loss reduction in a neural network model

Question

The following code is to train a neural network model of a given dataset (50,000 samples, 64 dim).

from keras import Sequential
from keras.layers import Dense
from keras.optimizers import Adam


X, y = process_dataset()
model = Sequential([
    Dense(16, input_dim=X.shape[1], activation='relu'),
    Dense(16, activation='relu'),
    Dense(1, activation='sigmoid')
])

'''
    Compile the Model
'''
model.compile(loss='binary_crossentropy', optimizer=Adam(lr=0.01), metrics=['accuracy'])

'''
    Fit the Model
'''
model.fit(X, y, shuffle=True, epochs=1000, batch_size=200, validation_split=0.2, verbose=2)

In the beginning, you can see below that the val_loss gets reduced from one epoch to another very well.

Epoch 82/1000
 - 0s - loss: 0.2036 - acc: 0.9144 - val_loss: 0.2400 - val_acc: 0.8885
Epoch 83/1000
 - 0s - loss: 0.2036 - acc: 0.9146 - val_loss: 0.2375 - val_acc: 0.8901

When the model takes many epochs, the loss change becomes so small, especially when the number of epochs increases.

Epoch 455/1000
 - 0s - loss: 0.0903 - acc: 0.9630 - val_loss: 0.1317 - val_acc: 0.9417
Epoch 456/1000
 - 0s - loss: 0.0913 - acc: 0.9628 - val_loss: 0.1329 - val_acc: 0.9443

Kindly, I have two questions:

1. What does this phenomenon mean? i.e., the loss begins to decrease very well at the beginning but not much reduction by the time the training epochs takes a lot of iteration.
2. What is the possible solution for this?

Thank you,

Answer 1

Generally, the decrease in loss tends to be smaller, the longer you train your model. You can think about this in a way, that the model first makes good progress in learning, but later any further improvement becomes harder (thus slower). At some point the model stops to learn. This comes from the logic of gradient decent, a numerical optimization process which is behind most ML models. If the model has learned what it is able to learn, loss does not decrease any more.

What can you do about this? You can try to make your model „better“ in terms of learning capacity. You can increase the capacity of the model (more neurons) or add more layers. You can also adjust the learning rate during learning by „callbacks“ (ReduceLROnPlateau). In this case, you lower the LR automatically if learning progress becomes small. By doing so you can try to make the model to learn more detailed patterns. See callbacks for Keras: https://keras.io/callbacks/.

Here is a nice blogpost about how to train NN: http://karpathy.github.io/2019/04/25/recipe/