Is improving a Neural Network really just "trial and error"?

Question

After asking on StackOverflow, I was redirected here, so I'm reposting this question.

I am a PhD student in Computational Physics and I've started to study a bit of Neural Networks, and decided to try and use some of what I've learned for a problem I'm having. After some studying, I've understood how to build a Neural Network for my purpose, but I can't find relevant info about how to build a good Neural Network apart from the good old trial and error. Here I attach the NN I'm currently working with as an example, but my question applies to the general case of a (regression) neural network: is there some theory on why I should build an architecture instead of another one, what activator I should choose, why I should lower my learning rate and how much, why should my dropout rate be higher and how much, how much training data is enough, and all these sorts of things?

My NN takes as input a 2x7 array of real values in [0,1] and gives as output a single real value, and it looks like this:

model_cnn = Sequential()
model_cnn.add(Conv2D(32, (2, 2), activation='relu', input_shape=(2, 7, 1), padding='same', kernel_regularizer=keras.regularizers.l2(0.01)))
model_cnn.add(BatchNormalization())
model_cnn.add(Conv2D(64, (2, 2), activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
model_cnn.add(BatchNormalization())
model_cnn.add(Flatten())
model_cnn.add(Dropout(0.5)) 
model_cnn.add(Dense(128, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
model_cnn.add(BatchNormalization())
model_cnn.add(Dropout(0.5)) 
model_cnn.add(Dense(64, activation='relu', kernel_regularizer=keras.regularizers.l2(0.01)))
model_cnn.add(BatchNormalization())
model_cnn.add(Dense(1, activation='linear')) #linear for regression

def lr_schedule(epoch):
    lr = 1e-3
    if epoch > 50:
        lr *= 0.1
    if epoch > 100:
        lr *= 0.1
    return lr

lr_scheduler = keras.callbacks.LearningRateScheduler(lr_schedule)
early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', 
                        patience=20, 
                        restore_best_weights=True)

model_cnn.compile(loss='mean_squared_error',
              optimizer=keras.optimizers.Adam(learning_rate=lr_schedule(0)),
              metrics=['mean_absolute_error'])

nn_history = model_cnn.fit(X_train, y_train,
                        batch_size=64,
                        epochs=1000,
                        verbose=1,
                        validation_data=(X_val, y_val),
                        callbacks=[lr_scheduler, early_stopping])

This is the result of some adjustments, for example adding dropout and normalization, that I did just by feeling, without any actual knowledge of, for example, whether it is correct to put them in the above order. Again: I know I can just try to change it and see what happens, but I'm asking how (if there is a way!) to decide what are the plausible, if not the best, things to do.
As of now, the loss-vs-epoch looks like this:

Doesn't look too bad, but I would like the loss to converge to zero (or at least to a value closer to zero). How can I understand what things are "worth trying"?

Answer 1

I'm new to this field as well, and from what I've learnt, yes, a lot of what we're doing while working with NNs is empirical.

We mostly build such NNs to work with some specific type of data, and since data comes in all sizes and shapes, there is no single specific set of rules to follow for the architecture of our network or the values of the hyper-parameters such that it yields best results regardless of the nature of data you're working with.

The trail and error part is necessary because you're trying to make a NN specifically for your use case.

We can of course learn from existing models and architectures. If you find a model that works with data similar to yours, you could read up on its architecture and why they decided to do things in that certain way. Might give you some insights on why a certain characteristic of the model's architecture might work for the type of data you're working with.

That being said, if you're feeling lost on what LR to use and why, which activation function to use and why, or why use regularization and where, then what you may be missing is a foundational understanding in these concepts.

I would recommend having a look at below two resources:

Andrew Ng's Deep Learning Specialization: https://www.coursera.org/specializations/deep-learning

Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville https://www.deeplearningbook.org/

Good luck.

Answer 2

Some activation functions work better in some cases. Hidden layers with ReLU work surprisingly well.

I would usually recommend adding some regularization to generalize better. Use a variable learning rate and momentum. Early stopping could be useful too but may result in a worse loss.

Your loss seems to be pretty good tbh.

As for dropout google "Geoffrey Hinton's dropout paper" gives guidance on how to manage dropout.

It would help if you would mention the tasks you are trying to achieve. Conv layers are not typically used for regression. They help learn local patterns and are used mostly in images.