1
$\begingroup$

I am new to the whole ML scene and am trying to resolve the Allstate Kaggle challenge to get a better feeling for the Random Forest Regression technique.

The challenge is evaluated based on the MAE for each row.

I've run the sklearn RandomForrestRegressor on my validation set, using the criterion=mae attribute. To my understanding this will run the Forest algorithm calculating the mae instead of the mse for each node.

After that I've used this: metrics.mean_absolute_error(Y_valid, m.predict(X_valid)) in order to calculate the MAE for each row of data.

What I would like to know is if the logic I'm following is sound. Am I making a fundamental mistake or missing something here? Should I have used the default MSE based Regressor and then calculate the MAE of each row using the mean_absolute_error function?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Let me clarify few fundamental things:

  1. In sklearn, RandomForrest Regressor criterion is:

    The function to measure the quality of a split

    It's a performance measure (by default, MSE) which helps the algorithm to decide on a rule for an optimum split on a node in a tree.

  2. Kaggle is giving you a metric, i.e. MAE (again a performance/ quality measure) but to evaluate the performance of your ML model, once finalized.

To come back to your question: while both MAE/ MSE are performance measures, they are being used at two different stages of a modeling process and might not be related. So, while it makes sense to evaluate your final model on MAE as you would be judged on it, you can choose any of MAE/ MSE for criterion (i.e. for RandomForest) depending on performance at validation stage.

While the above being said, keep in mind that you might want to evaluate the validation errors (i.e. for finalizing a model) on the same metric (i.e. MAE in this case), to keep error measure consistent with the test set evaluation.

Improve this answer
$\endgroup$
5
  • $\begingroup$ So let me get this straight. The criterion can be either MSE or MAE depending on whether my model performs better in the validation data with the former or the latter. And in the end I must use the mean_absolute_error function to get the MAE score, regardless of the chosen criterion, right? $\endgroup$
    – kingJulian
    Jan 6, 2019 at 10:08
  • $\begingroup$ @kingJulian added some more clarity. see last para. $\endgroup$
    – Mankind_008
    Jan 6, 2019 at 16:16
  • $\begingroup$ Thanks @Mankind_008! One question- what would be a possible way of validating errors? $\endgroup$
    – kingJulian
    Jan 6, 2019 at 16:18
  • 2
    $\begingroup$ @kingJulian There are many ways to validate but the good thing about RF is that it has an inbuilt validation mechanism, all you need is to just get the MAE for the regression on train/ validation data from the predictions of your model. $\endgroup$
    – Mankind_008
    Jan 6, 2019 at 16:26
  • $\begingroup$ It may feel counter-intuitive at first, but RF models that "internally" uses the MSE criterion often perform as well if not better in terms of MAE validation than RF models that "internally" use the MAE criterion itself. Since the RF models that rely on "internal" MSE criterion run much faster than those who run on MAE (see stackoverflow.com/questions/57243267/…), it's worth giving a try to the MSE-based RF even if your final validation metric is MAE. $\endgroup$
    – FZS
    Dec 23, 2022 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.