Working on this Kaggle competition, and have some questions. Using this code:
def r2_keras(y_true, y_pred): SS_res = K.sum(K.square(y_true - y_pred)) SS_tot = K.sum(K.square(y_true - K.mean(y_true))) return ( 1 - SS_res/(SS_tot + K.epsilon()) )
The output of my training looks like:
0s 138us/step - loss: 0.1340 - mean_squared_error: 0.1340 - r2_keras: 0.7565 - val_loss: 0.4112 - val_mean_squared_error: 0.4112 - val_r2_keras: 0.4064 Scaled Validation r2: 0.5182 Unscaled Validation r2: -152.1261
I am using 20% of the training data for validation.
I am tracking these metrics during training:
- Training loss, mse and r2
- Validation loss, mse and r2
I get these metrics on the model after training:
R2 for Validation on scaled data
R2 for Validation on unscaled data
scaler = StandardScaler() scaled_train = scaler.fit_transform(train_df) scaled_test = scaler.transform(test_df) ... m.fit( X_train, Y_train, epochs=epochs, validation_data=(X_test,Y_test)) .... from sklearn.metrics import r2_score scaled_r2 = r2_score(prediction, scaled_test_df[[target]].values) unscaled_r2 = r2_score(descaled_prediction, test_df[target].values)
So, my questions are:
- Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?
- Why isn't the unscaled r2 the same as the scaled r2?
- The model r2 is not the same as any of the validation r2's during training (
val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?