Improve the loss reduction in a neural network model

The following code is to train a neural network model of a given dataset (50,000 samples, 64 dim).

from keras import Sequential
from keras.layers import Dense

X, y = process_dataset()
model = Sequential([
Dense(16, input_dim=X.shape[1], activation='relu'),
Dense(16, activation='relu'),
Dense(1, activation='sigmoid')
])

'''
Compile the Model
'''

'''
Fit the Model
'''
model.fit(X, y, shuffle=True, epochs=1000, batch_size=200, validation_split=0.2, verbose=2)


In the beginning, you can see below that the val_loss gets reduced from one epoch to another very well.

Epoch 82/1000
- 0s - loss: 0.2036 - acc: 0.9144 - val_loss: 0.2400 - val_acc: 0.8885
Epoch 83/1000
- 0s - loss: 0.2036 - acc: 0.9146 - val_loss: 0.2375 - val_acc: 0.8901


When the model takes many epochs, the loss change becomes so small, especially when the number of epochs increases.

Epoch 455/1000
- 0s - loss: 0.0903 - acc: 0.9630 - val_loss: 0.1317 - val_acc: 0.9417
Epoch 456/1000
- 0s - loss: 0.0913 - acc: 0.9628 - val_loss: 0.1329 - val_acc: 0.9443


Kindly, I have two questions:

1. What does this phenomenon mean? i.e., the loss begins to decrease very well at the beginning but not much reduction by the time the training epochs takes a lot of iteration.
2. What is the possible solution for this?

Thank you,

What can you do about this? You can try to make your model „better“ in terms of learning capacity. You can increase the capacity of the model (more neurons) or add more layers. You can also adjust the learning rate during learning by „callbacks“ (ReduceLROnPlateau). In this case, you lower the LR automatically if learning progress becomes small. By doing so you can try to make the model to learn more detailed patterns. See callbacks for Keras: https://keras.io/callbacks/.