1
$\begingroup$

I am filing this issue after being stagnated here for couple of weeks. I am using hyperas to find the hyperparameters for my network, Densenet. My issue here is that my evaluation always fails with bad_alloc issue after few evaluations, After much googling this seems to be a memory related issue. I have tried the following things

  1. decrease batchsize to 2 from 64 still fails
  2. use of K.clear_session() at the start of my create_model or at the end, still bad_alloc,
  3. I noticed the code fails after one evaluation is done and time to evaluate the result. So I decreased the evaluation batchsize to 4 from default 32, still fails

So anyone here have any idea what is the issue here and how can it be fixed?

my code as follows(using hyperas on Densenet):

def create_model(X_train,y_train,X_val,y_val,X_test,y_test):
    K.clear_session()
    epochs = {{choice([5,7,10])}}
    es_patience = 3
    lr_patience = 2
    dropout = {{uniform(0.1,0.5)}}
    depth = {{choice([28,31,34,25])}}
    nb_dense_block = {{choice([3,4])}}
    nb_filter = 16
    growth_rate = {{choice([12,18,24,30])}}
    bn = True
    reduction_ = 0.5
    bs = 32
    lr = {{choice([2E-4,1E-4,5E-4])}}
    weight_file = 'keras_dn_wt_16Oct2200.h5'
    nb_classes = 1
    img_dim = (2,96,96)
    n_channels = 2
    print("------------------------ current config for the test -------------------------")
    print("Depth: ",depth," Growth_rate: ",growth_rate," lr: ",lr," nb_filter: ",nb_filter," dropout: ",dropout)
    print("Epochs ",epochs," batch_size: ",bs," es_patience: ",es_patience," lr_patience: ",lr_patience)
    print("dense_block ",nb_dense_block," reduction_: ",reduction_," bottleneck: ",bn)
    print("------------------------   end of configs        -------------------------")

    model  = DenseNet(depth=depth, nb_dense_block=nb_dense_block,
                 growth_rate=growth_rate, nb_filter=nb_filter,
                 dropout_rate=dropout,activation='sigmoid',
                 input_shape=img_dim,include_top=True,
                 bottleneck=bn,reduction=reduction_,
                 classes=nb_classes,pooling='avg',
                 weights=None)](url)
    model.summary()
    opt = Adam(lr=lr)
    model.compile(loss=binary_crossentropy, optimizer=opt, metrics=['accuracy'])

    es = EarlyStopping(monitor='val_loss', patience=es_patience,verbose=1)
    checkpointer = ModelCheckpoint(filepath=weight_file,verbose=1, save_best_only=True)

    lr_reducer = ReduceLROnPlateau(monitor='val_loss', factor=np.sqrt(0.1), cooldown=0, patience=lr_patience, min_lr=0.5e-6,verbose=1)

    model.fit(X_train,y_train,
          batch_size=bs,
          epochs=epochs,
          callbacks=[lr_reducer,es,checkpointer],
          validation_data=(X_val,y_val),
          verbose=2)

    score, acc = model.evaluate(X_test,y_test)
    print('current test accuracy:', acc)
    pred = model.predict(X_test)
    auc_score = roc_auc_score(y_test,pred)
    print("current test auc_score ------------------> ",auc_score)
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy