0
$\begingroup$

I am training my model on almost 200 000 images, i'm using Jupyter and now after 3 days of training ( i used 800 epochs and batch-size = 600) I have this " the kernel appears to have died. It will restart automaticaly" And this appears after 143 epochs only. Can anyone help me to solve this, and also can anyone advise me something in case of using big amount of data, because i am struggling with this dataset and I can't retrain the model each time the Jupyter blocks. Infact, I'm working on my internship project so I have to use all the data. I will be so grateful for your help.

$\endgroup$
1
$\begingroup$

I have had the same problem while training in huge data sets in Jupyter Notebooks. The only solution I found was to create a scrip .py with my training process (including model persistence) and running it from my terminal (python3 myscript.py)

$\endgroup$
2
  • $\begingroup$ So I juste have to download my file from Jupyter and run it in my terminal ? $\endgroup$
    – Lema Zaidi
    May 10 at 7:19
  • $\begingroup$ Not as simple as that. What I meant Is to create a script .py or a set of scripts .py in which you will code all your process so that at the end you can call your file from terminal and this file will run, execute and train all what you need. So in this way you won't have to run it inside a notebook $\endgroup$
    – Moreno
    May 11 at 21:45
0
$\begingroup$

Depending on the library you use you should be able to create a checkpoint of your model every few iterations so that you dont lose your models in the event of a crash. If you are unlucky enough to encounter a crash, you can always begin retraining from the latest available checkpoint. That way you don't start from scratch. Good Luck on your internship.

$\endgroup$
3
  • $\begingroup$ Btw, if you are using keras, this features of saving model checkpoints is baked into the library as a callback. $\endgroup$ May 7 at 14:49
  • $\begingroup$ Ok, I will try that. BTW, I have 2000 classes and I'm using transfer learning with VGG16 for image classification. When I save my model I notice that the size of the file (.h5) is 203 Mo, don't you think that it is big or it can cause a very slow training ? I'm using a powerful machine but even when I use batche-size = 600 with 800 epochs it's very slow, it takes days for a small number of epochs. Thank you $\endgroup$
    – Lema Zaidi
    May 7 at 16:05
  • $\begingroup$ I tried training with checkpoints but sometimes it doesn't begin from the latest results that I got and the loss increases! should I use checkpoints with only one file or is it better to use multiple files for each improvement of accuracy? and should I compile the model after interrupting the training? I wish I get an answer. Thank you for helping $\endgroup$
    – Lema Zaidi
    May 10 at 11:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.