7
$\begingroup$

I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are continuous variables. So an evaluation box plot looks like this: Ground truth vs. predicted values

I experiment with both linear (linear regression, linear SVMs) and nonlinear models (SVMs with RBF, Random forest, Gradient boosting machines ). The models are trained using cross-validation (~1600 samples), and 25% of the dataset is used for testing (~540 samples). I am using R-squared and Root Mean Square Error (RSME) to evaluate the models on test samples. I am interested in finding an evaluation measure to compare linear models to nonlinear ones.

This is done for scientific research. It was pointed out that R-square might not be an appropriate measure for nonlinear models, and that the Chi-Square test would be a better measure for goodness of fit.

The problem is, I am not sure what is the best way to do it. When I browse Chi-square as the goodness of fit, I only get examples where the Chi-square test is used to see whether some categorical samples fit a theoretical expectation, such as here. So here are my considerations/questions:

  1. One way I could think of is to categorize predicted (continuous) values into bins, and compare predicted distribution to the ground truth distribution using the Chi-Square test. But that doesn't make much sense, i.e. we have a machine learning model that perfectly predicts ground truth values 2,3, and 4, and values 5 predicts as 1, and values 1 as 5 - Chi-Square test that I propose here would reject the null hypothesis, although the model is mispredicting 2 out of 5 values.

  2. As referred to in a tutorial from USC I could use formula (1) to compute Chi-Square value, where experimentally measured quantities (xi) are my ground truth values, and hypothesized values (mui) are my predicted values. My question is, what is the variance? If we observe each value 1,2,3,4, and 5 as a distinct category, then the variance of ground truth within each category is equaled to zero. Also, how one computes the degree of freedom (N-r)?

  3. Related to the statement I am interested in finding an evaluation measure to compare linear models to nonlinear is the Chi-Square test the best (or even good) choice? What I've seen so far in machine learning competitions for regression tasks, either MSE or RSME are used for evaluation.

$\endgroup$
0
$\begingroup$

Use your test data to compare the predictive performance of each model.

In R you could do this like:

linear.predictions <- predict(linear.model, newdata = test.data)
nonlinear.predictions <- predict(nonlinear.model, newdata = test.data)

linear.percent.difference <- (test.data$TARGET_VARIABLE -
                              linear.predictions)  /
                              test.data$TARGET_VARIABLE

nonlinear.percent.difference <- (test.$TARGET_VARIABLE -
                                 nonlinear.predictions) /
                                 test.dtat$TARGET_VARIABLE

linear.grade <- mean(linear.percent.difference)
nonlinear.grade <- mean(nonlinear.percent.difference)

This is a pretty simple way to do it, but it is one that works for me and is easy to understand, especially if your audience is going to eye-glaze as soon as you say "Chi-square..." Get creative!

$\endgroup$
5
  • $\begingroup$ Thanks. Your proposal is unrelated to Chi-square test, right? What I can get from code, $modelGrade = 1/N\sum_{i=1}^{N} (y_i-yp_i)/y_i$, where y are ground truth values, and yp are values predicted by the machine learning model. Imagine a situation where we have two models predicting the same distribution of residuals (y-yp) – one makes more errors on examples with ground truth 1 and 2, and another one with ground truth values 4 and 5. So second one will have lower grade (and would be rated better?), although their predictive performance is the same. Sorry, but this does not make sense to me. $\endgroup$
    – Alex
    Aug 10 '18 at 20:29
  • $\begingroup$ Yes what I suggest is unrelated to Chi-square. I am suggesting calculating a percent difference between predicted and actual values. yp is a measure of the difference between predicted and actual. How large the difference is depends on the model, not the actual values. For example if y = 3 and Model 1 predicts 1 and Model 2 predicts 5 the difference is still 2. Are you suggesting the mean should be calculated on the absolute value of the percentages? This would keep negatives and positives from summing to 0 and skewing the result. I think that is a good idea. $\endgroup$
    – bstrain
    Aug 10 '18 at 23:18
  • $\begingroup$ In my previous comment, I was trying to interpret your code, not suggesting anything. It looks like you are computing residuals (difference between real/ground truth values and predicted values), then dividing it with real/ground truth and then computing the mean of it. The division part is what I am bothered by (see my previous comment in which I try to give an example explaining why your measure is not valid). Anyways, there are reliable evaluation measures for regression problems, for example check [Mean Square Error] (en.wikipedia.org/wiki/Mean_squared_error) $\endgroup$
    – Alex
    Aug 13 '18 at 18:54
  • $\begingroup$ You asked for additional metrics that could be interpreted across model types. What I have suggested is a metric that you can use. The division exists only to show the residual as a percentage to ease interpretability. $\endgroup$
    – bstrain
    Aug 13 '18 at 21:00
  • 1
    $\begingroup$ Yes, I have asked that, and so far you have been the only one to answer. I appreciate that, thanks. You intentions are good, but we just seem not to agree about the validity of the metric you propose. No offense :). $\endgroup$
    – Alex
    Aug 15 '18 at 20:19
0
$\begingroup$

You should frame your problem as ordinal regression. Then the model would predict target value, one of the five integer values.

As a result, the evaluation would not be best done with Root Mean Square Error (RSME). Chi-Square test could be applied between expected and predict counts for each of the five value levels.

If you want to then add in other model types, find the ordinal analogs (ordinal SVM or ordinal decision tree). The same Chi-Square test based on counts can be applied to find the best model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.