What is the num_initial_points argument for Bayesian Optimization with Keras Tuner?

Question

I've implemented the following code to run Keras-Tuner with Bayesian Optimization:

def model_builder(hp):
    
    NormLayer = Normalization()
    NormLayer.adapt(X_train)
    
    model = Sequential()

    model.add(Input(shape=X_train.shape[1:]))

    model.add(NormLayer)
    
    for i in range(hp.Int('conv_layers',2,4)):
        model.add(Conv1D(hp.Choice(f'kernel_{i}_nr',values=[16,32,64]), hp.Choice(f'kernel_{i}_size',values=[3,6,12]), strides=hp.Choice(f'kernel_{i}_strides',values=[1,2,3]), padding="same"))
        model.add(BatchNormalization(renorm=True))
        model.add(Activation('relu'))
        model.add(MaxPooling1D(2,strides=2, padding="valid"))

    model.add(Flatten())
    model.add(Dropout(hp.Choice('dropout_flatten',values=[0.0,0.25,0.5])))
    
    for i in range(hp.Int('dense_layers',1,2)):
        model.add(Dense(hp.Choice(f'dense_{i}_size',values=[500,1000])))
        model.add(Activation('relu'))
        model.add(Dropout(hp.Choice(f'dropout_{i}_others',values=[0.0,0.25,0.5])))

    model.add(Dense(hp.Choice('dense_size_last',values=[100,200])))
    model.add(Activation('relu'))

    model.add(Dense(2))
    model.add(Activation('softmax'))
   
    opt = Adam(learning_rate=lrn_rate_init)
    
    earlystop = EarlyStopping(monitor='val_loss',patience=8,restore_best_weights=True)

    model.compile(loss='categorical_crossentropy',optimizer=opt,metrics=['accuracy'])
    
    return model

tuner = BayesianOptimization(model_builder,objective='val_loss',num_initial_points=??,max_trials=tuner_trials,directory='BayesianOptimization/',project_name='BayesianOptimization')

What do the num_initial_points argument does exactly and what should I set it to in my case?

Reading the documentation I see the description

The number of randomly generated samples as initial training data for Bayesian optimization

but not being an expert I don't exactly get what it means and how it will impact the optimization process.

Answer 1

The Bayesian optimization algorithm selects points to test based on a balance between exploring uncertain regions and exploiting high-performing regions. But before you've tested very many points, there's not much information to go on. So, in this implementation you can specify a number of completely-at-random points to evaluate to start, and after that the actual Bayesian exploration begins.

Setting a high number of random points gives you guaranteed "exploration" points; indeed, in the documentation for the package bayesian-optimization, we find:

init_points: How many steps of random exploration you want to perform. Random exploration can help by diversifying the exploration space.

(The default is 5 in that package, and 3 times the dimension in keras-tuner.) That said, you can also make the algorithm focus more or less on exploration/exploitation directly, using the beta parameter (kappa in bayesian-optimization, see this example notebook).