How to verify if the behavior of CNN model is correct?

Question

I am exploring using CNNs for multi-class classification. My model details are:

and the training/testing accuracy/loss:

As you can see from the image, the accuracy jumped from 0.08 to 0.39 to 0.77 to 0.96 in few epochs. I have tried changing the details of the model (number of filters, kernel size) but I still note the same behavior and I am not experienced in deep learning. Is this behavior acceptable? Am I doing something wrong? To give some context. My dataset contains power traces for a side channel attack on EdDSA implementation. Each trace has 1000 power readings.

Answer 1

Do you have the same results in the next epochs?

If yes, your learning rate might be too high: Are you using an Adam optimizer?

It could also happen with some other hyper parameters like:

• A too high dropout that resets too much your neural network. If it is set to 0.5 or more, you could try with a lower value like 0.1 or 0.2.

• A bad weigh initialization (use random or Xavier for good results for instance).

In order to be more specific, I would need to read part of the code.

Answer 2

Without further information, I would say it is not rare to see this kind of behavior. Sometimes a DNN's score jumps up and down especially in early training phase, what really matters is the evaluation score in the end.

If you are curious, try shuffle/split your data with another setting and train again. Chances are some batches of training data is heavily skewed towards a class thus giving some turbulence.

Answer 3

Having a model that converges quickly isn't necessarily a problem. It may be that there is a strong, easily detected relationship between your predictors and the target. There's nothing obviously wrong with your model, but plotting the model using keras.utils.plot_model(model) will help confirm the layers are connected correctly.

What seems more concerning is that your validation accuracy is higher than your training accuracy for these epochs. This may indicate a problem with how you've selected your validation data; it may not be independent of your training data, or it may not be a representative sample. So that's the first thing I'd check.

If you want the model training to converge more slowly, try reducing the learning rate (set or change the learning_rate optimizer parameter on your model.compile call).

If you want to get more information about the model around epochs 7-9, there are two ways to do this:

1. Use checkpointing, which will save your model at each checkpoint. You can then test the model at those checkpoints - either using your validation or test data to get more information about the predictions being made. (Set the checkpoint callback on your model.fit call).

2. Stop training at an epoch, again run some tests, then resuming training. To do this, you call model.fit multiple times. On the first call, set epochs=7, so the model trains for 7 epochs. Then check your model - you can check the weights and evaluate your test data. Then to continue training, call model.fit again but set initial_epoch=7 and epochs=8. Then your model will train for another epoch. Repeat this to stop at all the epochs you want to investigate, then for the last call to model.fit set epochs=200 to complete training.