I am having trouble sometimes getting a fit on my data, and when I restart fit (with shuffle=true) then I sometimes get a good fit.

See my previous question:

Why does my model sometimes not learn well from same data?

As a work around, I want to automatically restart the fitting process, if loss is high after x epochs. How can I achieve this?

Here is a simplified structure:

def train_till_good():
    # Some loop around train()

def train():
    model = VerySimpleNet2(); 

    checkpoint = keras.callbacks.ModelCheckpoint(filepath=images_root + dataset_name + '\\CheckPoint.hdf5')

    myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)

    model.compile(optimizer=myOpt, loss='categorical_crossentropy',  metrics=['accuracy'])

   LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)

    tensorboard = keras.callbacks.TensorBoard(log_dir='C:\\Tensorflow', histogram_freq=0,write_graph=True, write_images=False)

    model.fit(train_images, train_labels, shuffle=True, epochs=num_epochs,
              validation_data = (test_images, test_labels)

def VerySimpleNet2():
    model = keras.Sequential([
        keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    return model


Here is another question that describes the data:

Why does adding random pixels stop my model learning in cnn?


1 Answer 1


While you should be able to do that one way or another, it won't be a good solution to your problem described here and the other question. If the difference between runs are very high - ie it depends how you split your dataset, that probably means that there is something inherently wrong with the dataset or the way it is split into training and validation sets.

Somehow you might be able to get a good result on training and validation data and take that as your base model and use for making predictions, but that does not mean that it is the best result or even an accurate description of your model. For this purpose, you need to spare a third dataset to check the accuracy: splitting the entire dataset as (training/validation/test)

I know this does not actually answer your question, but I thought this would help your process.

  • $\begingroup$ I agree. I would prefer not to do this. However, I do not understand what might be wrong with my data. As I understand it, "shuffle" only shuffles Training data - I don't believe it shuffles across Training and Validation. As I provide my model with the 2 sets of data. Do you agree? $\endgroup$
    – ManInMoon
    Nov 5, 2019 at 12:28
  • $\begingroup$ Shuffle parameter of the fit method just shuffles training data before each epoch. Maybe a problem with the way the data is split into test and training? It might be easier to comment if you also shared parts of your data $\endgroup$
    – serali
    Nov 5, 2019 at 12:36
  • $\begingroup$ How can I share the data? $\endgroup$
    – ManInMoon
    Nov 5, 2019 at 13:23
  • $\begingroup$ I mean, you do not need to share the data if it is not public already, but just add a brief description of it in your question. Also perhaps how you split the data. And the way you label the data from previous question is not very clear. All in all, you can describe the raw data and the preprocessing steps you have taken including labeling. $\endgroup$
    – serali
    Nov 5, 2019 at 13:48
  • $\begingroup$ See EDIT 1. I described the data in an earlier question. But basically I am filling in pixels randomly, with Green or Red. Except for a single pixel in top-half of image which I use to label the image. This is a completely artificial project I have created to test the fitting process - before using real (but very similar) images. $\endgroup$
    – ManInMoon
    Nov 5, 2019 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.