2
$\begingroup$

I have been exploring different regularization approaches and observed the most common to be using either Dropout Layers or L1/L2 Regularization. I have seen many debates of whether it is of interest to either combine or seperate regularization methods.

In my case I have implemented/integrated both approaches (combined and separate). For which I have seen promising results when actually combining as it has helped me not to always overfit my models entirely while generally improving the r2 score of my model.

Question:

Is it preferable to combine L1/L2 Regularization with Dropout Layer, or is it better to use them separately?

Example Code:

def model_build(x_train):
    # Define Inputs for ANN
    input_layer = Input(shape = (x_train.shape[1],), name = "Input")
    #Create Hidden ANN Layers
    dense_layer = BatchNormalization(name = "Normalization")(input_layer)  
    dense_layer = Dense(128, name = "First_Layer", activation = 'relu', kernel_regularizer=regularizers.l1(0.01))(dense_layer)
    #dense_layer = Dropout(0.08)(dense_layer)
    dense_layer = Dense(128, name = "Second_Layer", activation = 'relu',  kernel_regularizer=regularizers.l1(0.00))(dense_layer)
    #dense_layer = Dropout(0.05)(dense_layer)

    #Apply Output Layers
    output = Dense(1, name = "Output")(dense_layer)

    # Create an Interpretation Model (Accepts the inputs from branch and has single output)
    model = Model(inputs = input_layer, outputs = output)

    # Compile the Model
    model.compile(loss='mse', optimizer = Adam(lr = 0.01), metrics = ['mse'])
    #model.compile(loss='mse', optimizer=AdaBound(lr=0.001, final_lr=0.1), metrics = ['mse'])
$\endgroup$
2
$\begingroup$

I am unsure there will be a formal way to show which is best in which situations as it depends on many factors like your dataset, architecture of the your ANN - simply trying out different combinations is likely best.

It is worth noting that Dropout* is actually doing more than just regularization, it makes the model more robust,allowing it to try different nodes for prediction.

As for L1/L2 it just reduces overfitting by penalizing the higher weights.

$\endgroup$
2
  • $\begingroup$ that's alright. I'm just curious if having both approaches present in the model architecture would overwrite one method over the other. $\endgroup$ – machine_apprentice Oct 30 '20 at 10:34
  • 1
    $\begingroup$ Even though they both reduce overfitting, their approach is vastly different. They will not overwrite each other. $\endgroup$ – Shiv Oct 30 '20 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.