I want to judge the goodness of my neural network regression model built using Keras Python Library. The problem is the following: from an input like (1000, 5000) so 1000 samples and each sample has 5000 numbers, I want to predict, for each sample, 4 parameters that have generated each sample (each sample has its own 4 parameters). So the output is (1000,4). The 4 paramateres are generated from a Gamma distribution (in particular from
scipy.stats.gamma.rvs = (a=3, loc=0, scale=0.1, size=4, random_state=None)).
I splitted my data considering 5% of my data as test set and 95% as train set. So I have 50 data as test set and 950 as train set.
When I caculate the R squared for train and test data, should I consider the same number for both ? I mean that should I calculate the R squared for 50 test data and for 50 train data ?
Because if I calculate:
prediction_test = model.predict(X_test) print(r2_score(y_test, prediction_test))
I obtain R squared equal to 0.05916183503859888. While instead if I calculate R square on all train data (so 950 data) using the same python code:
prediction_train = model.predict(X_train) print(r2_score(y_train, prediction_train))
I obtain -6.939172963182204e+83 that is something of extremely bad result. While instead if I calculate the R squared on the same number of train data as the test set (so 50) I obtain 0.0751059501685121 and so a result more reasonable since the R squared of train data should be higher than the one calculated for test data because the model has been trained to recognize train data while instead the neural network has never seen test data. Is it right ?
So when comparing the train and test R squared, I should consider the same number of data for them ?
Any advice would be really appreciated and I thank you in advance.