-2
$\begingroup$

I am using a wrapper to use sklearn k-fold cross-validation with keras for a regression problem with ANN. but the accuracies i get look very weird. It has worked fine for a classification problem. I am attaching the code too. Is there anything I'm doing wrong

from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from keras.models import Sequential
from keras.layers import Dense
def build_regressor():
    regressor = Sequential()
    regressor.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu', input_dim = 15))
    regressor.add(Dense(units = 8, kernel_initializer = 'uniform', activation = 'relu'))
    regressor.add(Dense(units = 1, kernel_initializer = 'uniform'))
    regressor.compile(optimizer = 'adam', loss = 'mse', metrics = ['mae'])
    return regressor
regressor = KerasRegressor(build_fn = build_regressor, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = regressor, X = X_train, y = y_train, cv = 10, n_jobs = 1)
mean = accuracies.mean()
variance = accuracies.std()
$\endgroup$
5
  • $\begingroup$ What exactly do you mean they "look very weird"? Care to share them? $\endgroup$
    – desertnaut
    Mar 7 '19 at 1:41
  • $\begingroup$ -15.8012, -13.6942, -14.537, -22.315, -13.333, -15.8931, -16.9658, -13.4334, -21.4675, -39.7934, these are the 10 values obtained for accuracies $\endgroup$
    – cvg
    Mar 7 '19 at 6:52
  • $\begingroup$ "Accuracies" is the wrong term here (you are in a regression setting); so these are 10 values of negative MSE (or MAE). What is weird about them? $\endgroup$
    – desertnaut
    Mar 7 '19 at 8:20
  • $\begingroup$ I was expecting that "Accuracies" would contain r2_scores,since it is a regression problem. Correct me if I am wrong $\endgroup$
    – cvg
    Mar 7 '19 at 9:03
  • $\begingroup$ Well, the API is rather poorly documented, but I would be highly surprised if the Keras people use R^2 at all, which is practically never used in predictive contexts; R^2 seems like a fossil from the old statistics era - see the last part of my SO answer scikit-learn & statsmodels - which R-squared is correct? for more. $\endgroup$
    – desertnaut
    Mar 7 '19 at 9:55
1
$\begingroup$

Found the answer through sklearn documentation. The default scoring parameter for cross_val_score is None.So the accuracies that I got are not r2_scores. Since I was expecting them to be r^2 values, I have to mention it as a parameter.

accuracies = cross_val_score(estimator = regressor, X = X_train, y = y_train,scoring='r2',cv = 10, n_jobs = 1)

Adding scoring parameter I was able to get the r2_scores

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.