0
$\begingroup$

I've implemented the following code to run Keras-Tuner with Bayesian Optimization:

def model_builder(hp):
    
    NormLayer = Normalization()
    NormLayer.adapt(X_train)
    
    model = Sequential()

    model.add(Input(shape=X_train.shape[1:]))

    model.add(NormLayer)
    
    for i in range(hp.Int('conv_layers',2,4)):
        model.add(Conv1D(hp.Choice(f'kernel_{i}_nr',values=[16,32,64]), hp.Choice(f'kernel_{i}_size',values=[3,6,12]), strides=hp.Choice(f'kernel_{i}_strides',values=[1,2,3]), padding="same"))
        model.add(BatchNormalization(renorm=True))
        model.add(Activation('relu'))
        model.add(MaxPooling1D(2,strides=2, padding="valid"))

    model.add(Flatten())
    model.add(Dropout(hp.Choice('dropout_flatten',values=[0.0,0.25,0.5])))
    
    for i in range(hp.Int('dense_layers',1,2)):
        model.add(Dense(hp.Choice(f'dense_{i}_size',values=[500,1000])))
        model.add(Activation('relu'))
        model.add(Dropout(hp.Choice(f'dropout_{i}_others',values=[0.0,0.25,0.5])))

    model.add(Dense(hp.Choice('dense_size_last',values=[100,200])))
    model.add(Activation('relu'))

    model.add(Dense(2))
    model.add(Activation('softmax'))
   
    opt = Adam(learning_rate=lrn_rate_init)
    
    earlystop = EarlyStopping(monitor='val_loss',patience=8,restore_best_weights=True)

    model.compile(loss='categorical_crossentropy',optimizer=opt,metrics=['accuracy'])
    
    return model

tuner = BayesianOptimization(model_builder,objective='val_loss',num_initial_points=??,max_trials=tuner_trials,directory='BayesianOptimization/',project_name='BayesianOptimization')

What do the num_initial_points argument does exactly and what should I set it to in my case?

Reading the documentation I see the description

The number of randomly generated samples as initial training data for Bayesian optimization

but not being an expert I don't exactly get what it means and how it will impact the optimization process.

$\endgroup$
1
$\begingroup$

The Bayesian optimization algorithm selects points to test based on a balance between exploring uncertain regions and exploiting high-performing regions. But before you've tested very many points, there's not much information to go on. So, in this implementation you can specify a number of completely-at-random points to evaluate to start, and after that the actual Bayesian exploration begins.

Setting a high number of random points gives you guaranteed "exploration" points; indeed, in the documentation for the package bayesian-optimization, we find:

init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.

(The default is 5 in that package, and 3 times the dimension in keras-tuner.) That said, you can also make the algorithm focus more or less on exploration/exploitation directly, using the beta parameter (kappa in bayesian-optimization, see this example notebook).

$\endgroup$
2
  • $\begingroup$ Let me see if I understand this: if I have 50 initial points and 50 trials, it will be the same as RandomSearch? $\endgroup$ – WVJoe Feb 17 at 2:27
  • $\begingroup$ @WVJoe That's my understanding, yes. But now I see that I've been referencing bayesian-optimization, not the question's keras-tuner version. I'll edit my answer a bit. $\endgroup$ – Ben Reiniger Feb 17 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.