What metrics determine the quality of the model?

Question

Working on this Kaggle competition, and have some questions. Using this code:

def r2_keras(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true - y_pred)) 
    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) 
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

The output of my training looks like:

0s 138us/step - loss: 0.1340 - mean_squared_error: 0.1340 - r2_keras: 0.7565 - val_loss: 0.4112 - val_mean_squared_error: 0.4112 - val_r2_keras: 0.4064
Scaled Validation r2: 0.5182
Unscaled Validation r2: -152.1261

I am using 20% of the training data for validation.

I am tracking these metrics during training:

• Training loss, mse and r2
• Validation loss, mse and r2

I get these metrics on the model after training:

R2 for Validation on scaled data
R2 for Validation on unscaled data

scaler = StandardScaler()
scaled_train = scaler.fit_transform(train_df)
scaled_test  = scaler.transform(test_df)
...
m.fit( X_train, Y_train, epochs=epochs, validation_data=(X_test,Y_test))
....
from sklearn.metrics import r2_score
scaled_r2 = r2_score(prediction, scaled_test_df[[target]].values)
unscaled_r2 = r2_score(descaled_prediction, test_df[target].values)

So, my questions are:

1. Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?
2. Why isn't the unscaled r2 the same as the scaled r2?
3. The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?

Answer 1

Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?

I don't think this is a matter of which will describe the generalization error better, because both of them are describing the same thing, just on different scales. So, the advice would be to use the accuracy metric consistent with the metric that will be used for predictions on unseen data.

Why isn't the unscaled r2 the same as the scaled r2?

This is because MSE is scale dependent.

The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?

Why do you think so?

They are different because the datasets for training and for validation are different.