I would like to check if my understanding of how dropout layers should be used in Keras training is correct. I am training pretty simple MLP regression models:
#hidden layers model.add(Dense(6 ,input_dim=inputs_count ,activation='tanh')) model.add(Dropout(0)) model.add(Dense(8, activation='tanh')) model.add(Dropout(0.25)) model.add(Dense(4, activation='tanh')) #output neuron model.add(Dense(1, activation = 'linear')) model.compile(optimizer='adam', loss='mean_squared_error') es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3) model.fit(train_data[:, :-1], train_data[:, -1], batch_size=125, epochs=100, validation_split = 0.35, callbacks=[es], workers=8, use_multiprocessing=1)
I have always thought that models trained with dropout regularization should provide with similiar or slightly better results when predicting values from test data. But in my case network trained with dropout rate set from 0.25 to 0 returns significantly better results. I checked: net with $(1-0.25)\cdot8=6$ neurons in that layer provided good performance too. I am wondering why is that, and here are my conclusions:
- Layer size of 8 is perhabs too small for dropout rate 0.25? (in Keras it means a probability for dropping a connection, not retaining it)
- Maybe Adam optimiser with defauly parameters doesn't work properly with dropout -> still updating the weights which are turned to 0 by dropout? Should I switch to any other optimizer or use some adjusted parameters: greater/lesser learning rate no-momentum or greater momentum?
- There is something else in Keras implementation of Dropout and I am doing it in a generally wrong way or I missed some best practises using dropout so I am making some newbie mistakes.
If anyone have an idea what can I do to make better use of dropout, please let me know, because this case intrigues me!