1
$\begingroup$

I want to judge the goodness of my neural network regression model built using Keras Python Library. The problem is the following: from an input like (1000, 5000) so 1000 samples and each sample has 5000 numbers, I want to predict, for each sample, 4 parameters that have generated each sample (each sample has its own 4 parameters). So the output is (1000,4). The 4 paramateres are generated from a Gamma distribution (in particular from scipy.stats.gamma.rvs = (a=3, loc=0, scale=0.1, size=4, random_state=None)).

I splitted my data considering 5% of my data as test set and 95% as train set. So I have 50 data as test set and 950 as train set.

When I caculate the R squared for train and test data, should I consider the same number for both ? I mean that should I calculate the R squared for 50 test data and for 50 train data ?

Because if I calculate:

prediction_test = model.predict(X_test)
print(r2_score(y_test, prediction_test))

I obtain R squared equal to 0.05916183503859888. While instead if I calculate R square on all train data (so 950 data) using the same python code:

prediction_train = model.predict(X_train)
print(r2_score(y_train, prediction_train))

I obtain -6.939172963182204e+83 that is something of extremely bad result. While instead if I calculate the R squared on the same number of train data as the test set (so 50) I obtain 0.0751059501685121 and so a result more reasonable since the R squared of train data should be higher than the one calculated for test data because the model has been trained to recognize train data while instead the neural network has never seen test data. Is it right ?

So when comparing the train and test R squared, I should consider the same number of data for them ?

Any advice would be really appreciated and I thank you in advance.

Improve this question
$\endgroup$
13
  • $\begingroup$ You have four outputs. How does $R^2$ even make sense for that? // Even in a setting where $R^2$ makes sense, the function you’re using has minimal theoretical basis and probably isn’t telling you what you want it to tell you. What do you want to learn about your model by looking at $R^2?$ // There is no reason to care about the sample size, but your results are so bizarre that something odd is happening inside your code. What loss is your network minimizing? $\endgroup$
    – Dave
    Dec 26, 2022 at 11:53
  • $\begingroup$ I do not understand why $R^{2}$ does not make sense... I use it to compare the true parameters with those predicted by the model and so to understand if the model makes good predictions... Anyway thanks for the comment and I am using mae as loss function. $\endgroup$
    – HelpNeederStudent
    Dec 26, 2022 at 14:01
  • $\begingroup$ How would you calculate $R^2$ in your situation? What is the math, and why does that make sense? // Also, why look at $R^2$ if you are minimizing absolute loss? $\endgroup$
    – Dave
    Dec 26, 2022 at 14:19
  • $\begingroup$ The math is this scikit-learn.org/stable/modules/generated/… and this en.wikipedia.org/wiki/Coefficient_of_determination . You have the 4 parameters predicted by the model and the 4 true parameters. You apply the $R^{2}$ formula for all the test set (for example). My case is the case of a multi-output regression model so I choose to make an average between all the $R^{2}$ computed like the example in the scikit-learn page that I linked (where instead they computed a variance weighted). $\endgroup$
    – HelpNeederStudent
    Dec 26, 2022 at 14:28
  • $\begingroup$ Why are absolute loss function and $R^{2}$ not compatible ? @Dave $\endgroup$
    – HelpNeederStudent
    Dec 26, 2022 at 14:31

1 Answer 1

Reset to default
1
$\begingroup$

You calculate the values and compare them, just like you would for any other metric like MSE. Note, however, that the usual Python implementation of $R^2$ has poor theoretical motivation and probably does not tell you what you want it to tell you. (It certainly doesn’t tell me what I want it to tell me.)

Your strange results are occurring because you are trying to shoehorn an unusual problem into an inappropriate $R^2$ performance metric. If you use $R^2$ in a more reasonable setting, the results will make more sense.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks @Dave for your attention but I do not understand why $R^{2}$ in this context is inappropriate to use . If you look at the scikit-learn page there is also an example of application of $R^{2}$ to a case similar to mine (i.e. multi-output regression model) but they use as multioutput parameter the 'variance_weighted' instead of what I was using here that was ‘uniform_average’. $\endgroup$
    – HelpNeederStudent
    Dec 27, 2022 at 14:48
  • $\begingroup$ Now applying 'variance_weighted' as multi-output parameter to my data I obtain reasonable results... like $R^{2}_{test}=0.17$ and $R^{2}_{train}=0.23$ (by using 50 data to compute $R^{2}_{test}$ and all train data to compute $R^{2}_{train}$) $\endgroup$
    – HelpNeederStudent
    Dec 27, 2022 at 14:49
  • $\begingroup$ Maybe this is more suitable for my case ? Anyway now I read the post you linked. $\endgroup$
    – HelpNeederStudent
    Dec 27, 2022 at 14:50
  • $\begingroup$ so do you suggest to consider something like the MAE as a metrics of the goodness of my model in my "unusual" regression problem ? $\endgroup$
    – HelpNeederStudent
    Dec 28, 2022 at 8:01
  • $\begingroup$ I suggest thinking about what you’re trying to do and asking about appropriate ways to evaluate the performance of a model. Something like $R^2$ can make sense for a regression with multiple outputs, but your description seems to suggest a different kind of modeling approach than multivariate regression, more like estimating the conditional mean and conditional variance than estimating the conditional multivariate mean. $\endgroup$
    – Dave
    Dec 28, 2022 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.