0
$\begingroup$

My model:

    model = Sequential()
    model.add(Dense(128, activation='relu', input_dim=n_input_1))
    model.add(Dense(64, activation='relu'))
    #model.add(Dense(32, activation='relu'))
    #model.add(Dense(16, activation='relu'))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse',metrics=['mse'])

Now, I am doing hyper parameter tuning, but it is showing the same for every possible result:

Best: -61101.514139 using {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 15}

This is the first time I have done hyper parameter tuning before and this has stumped me. I can provide additional details if needed. What is a possible reason for this behavior?

I am doing time series forecasting using MLP. I have used 'neg_mean_absolute_error as the scoring function in gridsearchCV.

Edit: this is what Im running:

from sklearn.model_selection import GridSearchCV
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# define the grid search parameters
model = KerasClassifier(build_fn=create_model, verbose=1)
batch_size = [10,20,2000]
epochs = [2,4,5,10, 25]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3,scoring='neg_mean_squared_error')
grid_result = grid.fit(scaled_train,scaled_train_y)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))
$\endgroup$
  • $\begingroup$ Is the negative mean absolute error shown the error on the train or the validation set? $\endgroup$ – Oxbowerce Mar 13 at 22:40
  • $\begingroup$ Those are quite large values for mae; what's the scale of your output? As a first guess, perhaps the network just isn't learning anything at all. $\endgroup$ – Ben Reiniger Mar 14 at 2:18
  • $\begingroup$ @Oxbowerce i added the code $\endgroup$ – ubuntu_noob Mar 14 at 5:45
  • $\begingroup$ @BenReiniger from 100,000 to 200,000 $\endgroup$ – ubuntu_noob Mar 14 at 5:51
  • $\begingroup$ @ubuntu_noob what does the create_model function look like? $\endgroup$ – Oxbowerce Mar 14 at 9:47
2
$\begingroup$

I would suggest the following changes,

  1. Try KerasRegressor instead of KerasClassifier.
  2. Use other activation functions than relu (Eg: tanh, or just linear)
  3. Normalize both target and regressors.
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.