I am filing this issue after being stagnated here for couple of weeks. I am using hyperas to find the hyperparameters for my network, Densenet. My issue here is that my evaluation always fails with bad_alloc issue after few evaluations, After much googling this seems to be a memory related issue. I have tried the following things
- decrease batchsize to 2 from 64 still fails
- use of K.clear_session() at the start of my create_model or at the end, still bad_alloc,
- I noticed the code fails after one evaluation is done and time to evaluate the result. So I decreased the evaluation batchsize to 4 from default 32, still fails
So anyone here have any idea what is the issue here and how can it be fixed?
my code as follows(using hyperas on Densenet):
def create_model(X_train,y_train,X_val,y_val,X_test,y_test):
K.clear_session()
epochs = {{choice([5,7,10])}}
es_patience = 3
lr_patience = 2
dropout = {{uniform(0.1,0.5)}}
depth = {{choice([28,31,34,25])}}
nb_dense_block = {{choice([3,4])}}
nb_filter = 16
growth_rate = {{choice([12,18,24,30])}}
bn = True
reduction_ = 0.5
bs = 32
lr = {{choice([2E-4,1E-4,5E-4])}}
weight_file = 'keras_dn_wt_16Oct2200.h5'
nb_classes = 1
img_dim = (2,96,96)
n_channels = 2
print("------------------------ current config for the test -------------------------")
print("Depth: ",depth," Growth_rate: ",growth_rate," lr: ",lr," nb_filter: ",nb_filter," dropout: ",dropout)
print("Epochs ",epochs," batch_size: ",bs," es_patience: ",es_patience," lr_patience: ",lr_patience)
print("dense_block ",nb_dense_block," reduction_: ",reduction_," bottleneck: ",bn)
print("------------------------ end of configs -------------------------")
model = DenseNet(depth=depth, nb_dense_block=nb_dense_block,
growth_rate=growth_rate, nb_filter=nb_filter,
dropout_rate=dropout,activation='sigmoid',
input_shape=img_dim,include_top=True,
bottleneck=bn,reduction=reduction_,
classes=nb_classes,pooling='avg',
weights=None)](url)
model.summary()
opt = Adam(lr=lr)
model.compile(loss=binary_crossentropy, optimizer=opt, metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', patience=es_patience,verbose=1)
checkpointer = ModelCheckpoint(filepath=weight_file,verbose=1, save_best_only=True)
lr_reducer = ReduceLROnPlateau(monitor='val_loss', factor=np.sqrt(0.1), cooldown=0, patience=lr_patience, min_lr=0.5e-6,verbose=1)
model.fit(X_train,y_train,
batch_size=bs,
epochs=epochs,
callbacks=[lr_reducer,es,checkpointer],
validation_data=(X_val,y_val),
verbose=2)
score, acc = model.evaluate(X_test,y_test)
print('current test accuracy:', acc)
pred = model.predict(X_test)
auc_score = roc_auc_score(y_test,pred)
print("current test auc_score ------------------> ",auc_score)