How can I restart model fit after x epochs if loss is high?

Question

I am having trouble sometimes getting a fit on my data, and when I restart fit (with shuffle=true) then I sometimes get a good fit.

See my previous question:

Why does my model sometimes not learn well from same data?

As a work around, I want to automatically restart the fitting process, if loss is high after x epochs. How can I achieve this?

Here is a simplified structure:

def train_till_good():
    # Some loop around train()

def train():
    load_data()
    model = VerySimpleNet2(); 

    checkpoint = keras.callbacks.ModelCheckpoint(filepath=images_root + dataset_name + '\\CheckPoint.hdf5')

    myOpt = keras.optimizers.Adam(lr=0.001,decay=0.01)

    model.compile(optimizer=myOpt, loss='categorical_crossentropy',  metrics=['accuracy'])

   LRS = CyclicLR(base_lr=0.000005, max_lr=0.0003, step_size=200.)

    tensorboard = keras.callbacks.TensorBoard(log_dir='C:\\Tensorflow', histogram_freq=0,write_graph=True, write_images=False)

    model.fit(train_images, train_labels, shuffle=True, epochs=num_epochs,
              callbacks=[checkpoint,
                         tensorboard,
                         LRS],
              validation_data = (test_images, test_labels)
              )


def VerySimpleNet2():
    model = keras.Sequential([
        keras.layers.Dense(112, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
    return model

EDIT 1

Here is another question that describes the data:

Why does adding random pixels stop my model learning in cnn?

Answer 1

While you should be able to do that one way or another, it won't be a good solution to your problem described here and the other question. If the difference between runs are very high - ie it depends how you split your dataset, that probably means that there is something inherently wrong with the dataset or the way it is split into training and validation sets.

Somehow you might be able to get a good result on training and validation data and take that as your base model and use for making predictions, but that does not mean that it is the best result or even an accurate description of your model. For this purpose, you need to spare a third dataset to check the accuracy: splitting the entire dataset as (training/validation/test)

I know this does not actually answer your question, but I thought this would help your process.