0
$\begingroup$

I am training a CNN on some new dataset.

Usually, the accuracy steadily improves over 10-20 epochs.

I have created a new but similar dataset (using same methods) but now I see a sharp drop after 7th epoch, from which is never recovers.

What might this mean I am doing wrong?

3520/3662 [===========================>..] - ETA: 7s - loss: 0.1260 - acc: 0.9753
3552/3662 [============================>.] - ETA: 5s - loss: 0.1294 - acc: 0.9752
3584/3662 [============================>.] - ETA: 4s - loss: 0.1283 - acc: 0.9754
3616/3662 [============================>.] - ETA: 2s - loss: 0.1360 - acc: 0.9751
3648/3662 [============================>.] - ETA: 0s - loss: 0.1348 - acc: 0.9753
3662/3662 [==============================] - 199s 54ms/step - loss: 0.1387 - acc: 0.9752
Epoch 7/50

  32/3662 [..............................] - ETA: 3:29 - loss: 1.1921e-07 - acc: 1.0000
  64/3662 [..............................] - ETA: 3:44 - loss: 1.1921e-07 - acc: 1.0000
  96/3662 [..............................] - ETA: 3:43 - loss: 0.5037 - acc: 0.9688    
 128/3662 [>.............................] - ETA: 3:36 - loss: 0.6296 - acc: 0.9609
 160/3662 [>.............................] - ETA: 3:29 - loss: 0.8059 - acc: 0.9500
 192/3662 [>.............................] - ETA: 3:23 - loss: 0.7555 - acc: 0.9531
 224/3662 [>.............................] - ETA: 3:20 - loss: 0.7915 - acc: 0.9509
 256/3662 [=>............................] - ETA: 3:17 - loss: 0.6926 - acc: 0.9570
 288/3662 [=>............................] - ETA: 3:14 - loss: 0.7276 - acc: 0.9549
 320/3662 [=>............................] - ETA: 3:11 - loss: 0.9570 - acc: 0.9406
 352/3662 [=>............................] - ETA: 3:08 - loss: 1.0990 - acc: 0.9318
 384/3662 [==>...........................] - ETA: 3:05 - loss: 1.1531 - acc: 0.9271
 416/3662 [==>...........................] - ETA: 3:02 - loss: 1.1032 - acc: 0.9303
 448/3662 [==>...........................] - ETA: 3:00 - loss: 1.2832 - acc: 0.9174
 480/3662 [==>...........................] - ETA: 2:58 - loss: 1.3991 - acc: 0.9104
 512/3662 [===>..........................] - ETA: 2:55 - loss: 1.4691 - acc: 0.9062
 544/3662 [===>..........................] - ETA: 2:53 - loss: 1.4420 - acc: 0.9081
 576/3662 [===>..........................] - ETA: 2:50 - loss: 1.5018 - acc: 0.9045
 608/3662 [===>..........................] - ETA: 2:48 - loss: 1.4492 - acc: 0.9079
 640/3662 [====>.........................] - ETA: 2:46 - loss: 1.3768 - acc: 0.9125
 672/3662 [====>.........................] - ETA: 2:44 - loss: 1.3112 - acc: 0.9167
 704/3662 [====>.........................] - ETA: 2:42 - loss: 1.3431 - acc: 0.9148
 736/3662 [=====>........................] - ETA: 2:40 - loss: 1.3066 - acc: 0.9171
 768/3662 [=====>........................] - ETA: 2:38 - loss: 1.2941 - acc: 0.9180
 800/3662 [=====>........................] - ETA: 2:36 - loss: 1.3431 - acc: 0.9150
 832/3662 [=====>........................] - ETA: 2:34 - loss: 1.4464 - acc: 0.9087
 864/3662 [======>.......................] - ETA: 2:32 - loss: 1.5980 - acc: 0.8993
 896/3662 [======>.......................] - ETA: 2:30 - loss: 1.7568 - acc: 0.8895
 928/3662 [======>.......................] - ETA: 2:28 - loss: 2.0263 - acc: 0.8728
 960/3662 [======>.......................] - ETA: 2:27 - loss: 2.1434 - acc: 0.8656
 992/3662 [=======>......................] - ETA: 2:25 - loss: 2.3017 - acc: 0.8558
1024/3662 [=======>......................] - ETA: 2:23 - loss: 2.3557 - acc: 0.8525
1056/3662 [=======>......................] - ETA: 2:21 - loss: 2.5133 - acc: 0.8428
1088/3662 [=======>......................] - ETA: 2:20 - loss: 2.6468 - acc: 0.8346
1120/3662 [========>.....................] - ETA: 2:18 - loss: 2.8590 - acc: 0.8214
1152/3662 [========>.....................] - ETA: 2:16 - loss: 2.9475 - acc: 0.8160
1184/3662 [========>.....................] - ETA: 2:14 - loss: 3.1265 - acc: 0.8049
1216/3662 [========>.....................] - ETA: 2:13 - loss: 3.2430 - acc: 0.7977
1248/3662 [=========>....................] - ETA: 2:11 - loss: 3.3794 - acc: 0.7893
1280/3662 [=========>....................] - ETA: 2:09 - loss: 3.4964 - acc: 0.7820
1312/3662 [=========>....................] - ETA: 2:07 - loss: 3.5831 - acc: 0.7767
1344/3662 [==========>...................] - ETA: 2:05 - loss: 3.7257 - acc: 0.7679
1376/3662 [==========>...................] - ETA: 2:04 - loss: 3.8030 - acc: 0.7631
1408/3662 [==========>...................] - ETA: 2:02 - loss: 3.8883 - acc: 0.7578
1440/3662 [==========>...................] - ETA: 2:00 - loss: 4.0369 - acc: 0.7486
1472/3662 [===========>..................] - ETA: 1:58 - loss: 4.0587 - acc: 0.7473
1504/3662 [===========>..................] - ETA: 1:56 - loss: 4.2081 - acc: 0.7380
1536/3662 [===========>..................] - ETA: 1:55 - loss: 4.2673 - acc: 0.7344
1568/3662 [===========>..................] - ETA: 1:53 - loss: 4.4167 - acc: 0.7251
1600/3662 [============>.................] - ETA: 1:51 - loss: 4.4895 - acc: 0.7206
1632/3662 [============>.................] - ETA: 1:49 - loss: 4.5694 - acc: 0.7157
1664/3662 [============>.................] - ETA: 1:48 - loss: 4.6074 - acc: 0.7133
1696/3662 [============>.................] - ETA: 1:46 - loss: 4.6916 - acc: 0.7081
1728/3662 [=============>................] - ETA: 1:44 - loss: 4.7726 - acc: 0.7031
1760/3662 [=============>................] - ETA: 1:42 - loss: 4.8415 - acc: 0.6989
1792/3662 [=============>................] - ETA: 1:40 - loss: 4.8810 - acc: 0.6964
1824/3662 [=============>................] - ETA: 1:39 - loss: 4.9279 - acc: 0.6935
1856/3662 [==============>...............] - ETA: 1:37 - loss: 4.9558 - acc: 0.6918
1888/3662 [==============>...............] - ETA: 1:35 - loss: 4.9828 - acc: 0.6901
1920/3662 [==============>...............] - ETA: 1:33 - loss: 4.9921 - acc: 0.6896
1952/3662 [==============>...............] - ETA: 1:32 - loss: 4.9846 - acc: 0.6901
1984/3662 [===============>..............] - ETA: 1:30 - loss: 5.0423 - acc: 0.6865
2016/3662 [===============>..............] - ETA: 1:28 - loss: 5.1062 - acc: 0.6825
2048/3662 [===============>..............] - ETA: 1:26 - loss: 5.1602 - acc: 0.6792
2080/3662 [================>.............] - ETA: 1:25 - loss: 5.1970 - acc: 0.6769
2112/3662 [================>.............] - ETA: 1:23 - loss: 5.2022 - acc: 0.6766
2144/3662 [================>.............] - ETA: 1:21 - loss: 5.2449 - acc: 0.6740
2176/3662 [================>.............] - ETA: 1:20 - loss: 5.2937 - acc: 0.6710
2208/3662 [=================>............] - ETA: 1:18 - loss: 5.3337 - acc: 0.6685
2240/3662 [=================>............] - ETA: 1:16 - loss: 5.3295 - acc: 0.6687
2272/3662 [=================>............] - ETA: 1:14 - loss: 5.3538 - acc: 0.6673
2304/3662 [=================>............] - ETA: 1:13 - loss: 5.3843 - acc: 0.6654
2336/3662 [==================>...........] - ETA: 1:11 - loss: 5.4072 - acc: 0.6640
2368/3662 [==================>...........] - ETA: 1:09 - loss: 5.4022 - acc: 0.6643
2400/3662 [==================>...........] - ETA: 1:08 - loss: 5.4510 - acc: 0.6613
2432/3662 [==================>...........] - ETA: 1:06 - loss: 5.4853 - acc: 0.6591
2464/3662 [===================>..........] - ETA: 1:04 - loss: 5.5188 - acc: 0.6571
2496/3662 [===================>..........] - ETA: 1:02 - loss: 5.5707 - acc: 0.6538
2528/3662 [===================>..........] - ETA: 1:01 - loss: 5.5958 - acc: 0.6523
2560/3662 [===================>..........] - ETA: 59s - loss: 5.6455 - acc: 0.6492 
2592/3662 [====================>.........] - ETA: 57s - loss: 5.6691 - acc: 0.6478
2624/3662 [====================>.........] - ETA: 55s - loss: 5.7044 - acc: 0.6456
2656/3662 [====================>.........] - ETA: 54s - loss: 5.7631 - acc: 0.6419
2688/3662 [=====================>........] - ETA: 52s - loss: 5.7964 - acc: 0.6399
2720/3662 [=====================>........] - ETA: 50s - loss: 5.8467 - acc: 0.6368
2752/3662 [=====================>........] - ETA: 48s - loss: 5.8783 - acc: 0.6348
2784/3662 [=====================>........] - ETA: 47s - loss: 5.8918 - acc: 0.6340
2816/3662 [======================>.......] - ETA: 45s - loss: 5.9279 - acc: 0.6317
2848/3662 [======================>.......] - ETA: 43s - loss: 5.9405 - acc: 0.6310
2880/3662 [======================>.......] - ETA: 42s - loss: 5.9640 - acc: 0.6295
2912/3662 [======================>.......] - ETA: 40s - loss: 6.0037 - acc: 0.6271
2944/3662 [=======================>......] - ETA: 38s - loss: 6.0260 - acc: 0.6257
2976/3662 [=======================>......] - ETA: 36s - loss: 6.0370 - acc: 0.6250
3008/3662 [=======================>......] - ETA: 35s - loss: 6.0639 - acc: 0.6233
3040/3662 [=======================>......] - ETA: 33s - loss: 6.0743 - acc: 0.6227
3072/3662 [========================>.....] - ETA: 31s - loss: 6.0897 - acc: 0.6217
3104/3662 [========================>.....] - ETA: 29s - loss: 6.1048 - acc: 0.6208
3136/3662 [========================>.....] - ETA: 28s - loss: 6.1351 - acc: 0.6189
3168/3662 [========================>.....] - ETA: 26s - loss: 6.1443 - acc: 0.6184
3200/3662 [=========================>....] - ETA: 24s - loss: 6.1685 - acc: 0.6169
3232/3662 [=========================>....] - ETA: 23s - loss: 6.1822 - acc: 0.6160
3264/3662 [=========================>....] - ETA: 21s - loss: 6.1858 - acc: 0.6158
3296/3662 [==========================>...] - ETA: 19s - loss: 6.1991 - acc: 0.6150
3328/3662 [==========================>...] - ETA: 17s - loss: 6.2073 - acc: 0.6145
3360/3662 [==========================>...] - ETA: 16s - loss: 6.2729 - acc: 0.6104
3392/3662 [==========================>...] - ETA: 14s - loss: 6.2945 - acc: 0.6091
3424/3662 [===========================>..] - ETA: 12s - loss: 6.2875 - acc: 0.6095
3456/3662 [===========================>..] - ETA: 11s - loss: 6.3225 - acc: 0.6073
3488/3662 [===========================>..] - ETA: 9s - loss: 6.3431 - acc: 0.6061 
3520/3662 [===========================>..] - ETA: 7s - loss: 6.3633 - acc: 0.6048
3552/3662 [============================>.] - ETA: 5s - loss: 6.4058 - acc: 0.6022
3584/3662 [============================>.] - ETA: 4s - loss: 6.3980 - acc: 0.6027
3616/3662 [============================>.] - ETA: 2s - loss: 6.4217 - acc: 0.6012
3648/3662 [============================>.] - ETA: 0s - loss: 6.4537 - acc: 0.5992
3662/3662 [==============================] - 200s 55ms/step - loss: 6.4730 - acc: 0.5980
Epoch 8/50
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't know anything about the data you're using so I'll offer some suggestions.

Your learning rate might be too high. Maybe you're over-shooting the gradient descent such that after you approach a nice vector of coefficients, you're then jumping too far in one epoch and thus losing progress towards the "global optima".

Your activation function might be saturating your units.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am using a standard "un-trained" AlexNet CNN, and learning rate etc are the default Keras values. How can I check if I am saturating my units? $\endgroup$
    – ManInMoon
    Commented Apr 23, 2019 at 13:57
  • $\begingroup$ You can edit the default values. Lower the learning rate and try some of these activation functions: keras.io/layers/advanced-activations $\endgroup$
    – Sterls
    Commented Apr 23, 2019 at 14:07
  • $\begingroup$ You were right about the learning rate, that was the issue. Strange how default lr=0.01 was too fast. $\endgroup$
    – ManInMoon
    Commented Apr 26, 2019 at 15:41
  • $\begingroup$ Glad to hear it's $\endgroup$
    – Sterls
    Commented Apr 26, 2019 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.