Improving Accuracy of the Deep Learning Model

Question

In my current project, I have only 647 rows (500 for training and 147 for testing) and I have applied the Keras Sequential model using the following code:

from keras import models
from keras import layers
from keras import regularizers
model = models.Sequential()
model.add(layers.Dense(5,activation="relu",input_shape=(train_x.shape[1],)))
model.add(layers.Dense(1,activation="sigmoid"))
from keras import optimizers
#network = model.compile(optimizer=optimizers.RMSprop(lr=0.001),loss="binary_crossentropy",metrics=["accuracy"]) 
network = model.compile(optimizer=optimizers.Adam(lr=0.05, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False),loss="binary_crossentropy",metrics=["accuracy"]) 
result = model.fit(train_x,train_y,epochs=60, batch_size=32)

Following is the result of a few Epochs:

   Epoch 1/60
   500/500 [==============================] - 1s 3ms/step - loss: 0.7253 - acc: 0.5520
   Epoch 2/60 
   500/500 [==============================] - 0s 137us/step - loss: 0.6379 - acc: 0.6640
   Epoch 3/60
   500/500 [==============================] - 0s 134us/step - loss: 0.6035 - acc: 0.6880
   Epoch 4/60
   500/500 [==============================] - 0s 158us/step - loss: 0.5852 - acc: 0.6980
   Epoch 5/60
   500/500 [==============================] - 0s 136us/step - loss: 0.5864 - acc: 0.7140
   Epoch 6/60
   500/500 [==============================] - 0s 134us/step - loss: 0.5552 - acc: 0.7240
   Epoch 7/60
   500/500 [==============================] - 0s 141us/step - loss: 0.5475 - acc: 0.7280
   Epoch 8/60
   500/500 [==============================] - 0s 164us/step - loss: 0.5340 - acc: 0.7460
   Epoch 9/60
   500/500 [==============================] - 0s 138us/step - loss: 0.5389 - acc: 0.7280
   Epoch 10/60
   500/500 [==============================] - 0s 139us/step - loss: 0.5374 - acc: 0.7540
    ===================For the simplicity I am sharing first and last few epochs result=
   Epoch 55/60
   500/500 [==============================] - 0s 161us/step - loss: 0.4947 - acc: 0.7800
   Epoch 56/60
   500/500 [==============================] - 0s 168us/step - loss: 0.5058 - acc: 0.7660
   Epoch 57/60
   500/500 [==============================] - 0s 158us/step - loss: 0.5011 - acc: 0.7700
   Epoch 58/60
   500/500 [==============================] - 0s 154us/step - loss: 0.5062 - acc: 0.7660
   Epoch 59/60
   500/500 [==============================] - 0s 156us/step - loss: 0.5040 - acc: 0.7600
   Epoch 60/60
   500/500 [==============================] - 0s 147us/step - loss: 0.4994 - acc: 0.7800

Using the above configuration (I have also tried different Neural Network architecture, the above one looks fine), I am able to achieve the best accuracy so far and which is train accuracy ~ 78 % and test accuracy ~ 72 %. I also tried with Logistic regression but in this case train accuracy ~ 65 %.

Here it looks like overfitting occurs so I tried with L2 Regularization and Dropout but none of them help to achieve the better accuracy. Unfortunately, I can't generate more data.

What should I do to achieve the better accuracy of the model (given a limited amount of data)? How can I increase the efficiency of both train and test data?

Answer 1

You can try bunch of things like:

1. Data augmentation.
2. Transfer learning on model trained on similar dataset.
3. Train multiple neural network with different intialisation of weights and then take majority voting during inference.

The last point matters if you're training from scratch because weights intialisation direct the model towards different minima. So with average of different networks you're essentially taking average of different minimas which might increase overall accuracy.