0
$\begingroup$

I'm new to learning neural networks and I found an example online to test accuracy with k-fold cross validation.

The example is for binary data but I want to test MAE or RMSE (I guess?) for my regression prediction. I'm a bit lost now on how I can test it because currently the accuracy is mostly 0.00%, sometimes 1.94%, I assume because my data is not binary.

Here is my code:

seed = 7
numpy.random.seed(seed)
# load pima indians dataset
# split into input (X) and output (Y) variables
# define 10-fold cross validation test harness
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
cvscores = []
for train, test in kfold.split(X, Y):
    model = Sequential()
    model.add(Dense(32, input_dim=12, activation='relu'))
    model.add(Dense(16, activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='relu'))
    # Compile model
    model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['accuracy'])
    # Fit the model
    model.fit(X.iloc[train], Y.iloc[train], epochs=150, batch_size=10, verbose=0)
    # evaluate the model
    scores = model.evaluate(X.iloc[test], Y.iloc[test], verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
    cvscores.append(scores[1] * 100)
print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvscores), numpy.std(cvscores)))

So my question is: how do I test MAE for regression data with k-fold cross validation? Thanks for the help.

$\endgroup$

1 Answer 1

0
$\begingroup$

I don't think you should use accuracy as a metric, you can test your MAE (Mean Absolute Error) by changing the metric in the compiling line.

model.compile(..., metrics=['mae'])

If you want to use RMSE you can define it yourself (it's not built-in in keras):

from keras import backend
 
def rmse(y_true, y_pred):
    return backend.sqrt(backend.mean(backend.square(y_pred - y_true), axis=-1))

model.compile(loss='mse', optimizer='adam', metrics=[rmse])

Good luck :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.