0
$\begingroup$

I am exploring using CNNs for multi-class classification. My model details are:

model summary

and the training/testing accuracy/loss:

Training and validation accuracy/loss

As you can see from the image, the accuracy jumped from 0.08 to 0.39 to 0.77 to 0.96 in few epochs. I have tried changing the details of the model (number of filters, kernel size) but I still note the same behavior and I am not experienced in deep learning. Is this behavior acceptable? Am I doing something wrong? To give some context. My dataset contains power traces for a side channel attack on EdDSA implementation. Each trace has 1000 power readings.

$\endgroup$

3 Answers 3

1
+50
$\begingroup$

Do you have the same results in the next epochs?

If yes, your learning rate might be too high: Are you using an Adam optimizer?

It could also happen with some other hyper parameters like:

  • A too high dropout that resets too much your neural network. If it is set to 0.5 or more, you could try with a lower value like 0.1 or 0.2.

  • A bad weigh initialization (use random or Xavier for good results for instance).

In order to be more specific, I would need to read part of the code.

$\endgroup$
8
  • $\begingroup$ yes I am using Adam optimizer, is that a problem? Also I noticed that in my training dataset the samples are ordered according to the class ( for example the first 300 sample are class 0, the 2nd 300 sample are class 1 and so on) can this be a problem? $\endgroup$
    – Alaa
    May 26, 2022 at 13:41
  • $\begingroup$ You should randomize your data, and then split it in test and train sets. Try this first, before checking what could be done with Adam. $\endgroup$ May 26, 2022 at 21:03
  • $\begingroup$ I use train_test_split from sklearn which randomly splits the training and testing data $\endgroup$
    – Alaa
    May 27, 2022 at 11:26
  • $\begingroup$ Can you reduce the alpha (=learning rate) of the adam optimizer by 10 times less? $\endgroup$ May 27, 2022 at 11:58
  • 1
    $\begingroup$ Try with 0.1 dropout, it should work better. 0.5 or 50% dropout resets 50% of your neurons at each epoch, and hence prevent it from learning. $\endgroup$ May 27, 2022 at 13:07
2
$\begingroup$

Without further information, I would say it is not rare to see this kind of behavior. Sometimes a DNN's score jumps up and down especially in early training phase, what really matters is the evaluation score in the end.

If you are curious, try shuffle/split your data with another setting and train again. Chances are some batches of training data is heavily skewed towards a class thus giving some turbulence.

$\endgroup$
2
$\begingroup$

Having a model that converges quickly isn't necessarily a problem. It may be that there is a strong, easily detected relationship between your predictors and the target. There's nothing obviously wrong with your model, but plotting the model using keras.utils.plot_model(model) will help confirm the layers are connected correctly.

What seems more concerning is that your validation accuracy is higher than your training accuracy for these epochs. This may indicate a problem with how you've selected your validation data; it may not be independent of your training data, or it may not be a representative sample. So that's the first thing I'd check.

If you want the model training to converge more slowly, try reducing the learning rate (set or change the learning_rate optimizer parameter on your model.compile call).

If you want to get more information about the model around epochs 7-9, there are two ways to do this:

  1. Use checkpointing, which will save your model at each checkpoint. You can then test the model at those checkpoints - either using your validation or test data to get more information about the predictions being made. (Set the checkpoint callback on your model.fit call).

  2. Stop training at an epoch, again run some tests, then resuming training. To do this, you call model.fit multiple times. On the first call, set epochs=7, so the model trains for 7 epochs. Then check your model - you can check the weights and evaluate your test data. Then to continue training, call model.fit again but set initial_epoch=7 and epochs=8. Then your model will train for another epoch. Repeat this to stop at all the epochs you want to investigate, then for the last call to model.fit set epochs=200 to complete training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.