3
$\begingroup$

I am solving a problem with machine learning and I have some data with two integer type independent variables and a continuous dependent variable. I am optimising to RMSE. I had fairly large RMSE value on my validation data. I learned that my model didn't do good on larger values of target; so, I have tried removing rows with larger values and that didn't help. So, now in the process of understanding the mistakes, I calculated RMSE for each ground truth value and it's prediction from validation set and plotted it to understand where big mistakes had happened. Apparently, my model still doesn't do good at larger values of target.

Here is the plot: enter image description here

And here is the plot showing relationship between ground truth values and predictions: enter image description here As you can see, my model's predictions got worse as the values got large. How do I prevent this?

Some information about my data(only what I can reveal):

  1. There is absolutely no linear relationship between independent variables and target. So, I have used a tree based model and random forest is giving me relatively good results.
  2. I can even went to say that both of my independent variables can be called categorical variables with very high categories.
  3. Also, there are a lot of values in independent variables, that occur only once.
  4. all the variables are highly skewed to the right.(range of IV_1: 0 to 3,700; IV_2: 0 to 40; target variable: 0 to 39,000)

How do I decrease my RMSE or doing what will decrease it?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
3
$\begingroup$

It looks like that your prediction is clamping at 750.

Be mindful of the fact that Tree can't predict a Regression value that is outside the range it has been trained on.

So, first of all, please assure that your data doesn't have a trend.

Improve this answer
$\endgroup$
5
  • $\begingroup$ what do you mean by trend. I had a datetime variable and removing which actually improved predictive performance. $\endgroup$
    – Naveen Reddy Marthala
    Commented Nov 18, 2020 at 15:08
  • 1
    $\begingroup$ A trend means - Let's say your target is 0-5000. If you trained a Tree-based model on the dataset (0-2500), Val set - (2500-4000), and Test set as (4000-5000). Then your RMSE will increase with higher values(>2500) because the Tree will predict 2500 maximum. So your Test error will look very high (e.g. 1500 for 4000 and 2500 for 5000) $\endgroup$
    – 10xAI
    Commented Nov 18, 2020 at 15:14
  • $\begingroup$ No, I don't think I have such trend in my data. Also, you can see large concentration of values in the range (0-200) in my ground truths to predictions plot. And distribution of all variables, yes all independent variables, of my train and test sets match by more than 95 percent. $\endgroup$
    – Naveen Reddy Marthala
    Commented Nov 18, 2020 at 15:57
  • 1
    $\begingroup$ Could you please share you Ymax, Ymin for all the set i.e. Test/Val/Train $\endgroup$
    – 10xAI
    Commented Nov 19, 2020 at 16:17
  • $\begingroup$ train ,val and test has same range range, differs by insignificant amounts. for IV_1: 0 to 3,700; IV_2: 0 to 40; target variable: 0 to 39,000. you can see them in my question as well. The plot of predictions to ground truths of target you can see in my question has different range for predictions, because, those predictions were made by a model trained with outliers capped from the right. $\endgroup$
    – Naveen Reddy Marthala
    Commented Nov 19, 2020 at 16:50
1
$\begingroup$

If my understanding is right, you have a regression problem, with categorical features with high cardinality and "outliers" (or just big numbers).

How have you encoded categories? Target Encoding? There is another option that is not encoding with the mean but with the median that on some cases can perform better.

On this notebook , you can see an implementation adn the results of this method.

Improve this answer
$\endgroup$
3
  • $\begingroup$ yes, my problem is a regression problem. my IVs are numerical variables. just to give reader an idea, i said that values in those variables are repeated so many times, that it can be safe to call them categorical variables. I believe target encoding can't be done, since I have around 1.5% of numbers in my IVs in test data that are not present in train data. I haven't tried target encoding. $\endgroup$
    – Naveen Reddy Marthala
    Commented Nov 18, 2020 at 6:46
  • $\begingroup$ So you would have categories in test that are not in train do to real life constraints? Still in target encoding, you can handle how to work with unseen categories (mean, 0, median....) $\endgroup$
    – Carlos Mougan
    Commented Nov 18, 2020 at 7:02
  • $\begingroup$ sure @CarlosMougan, I will try them. Thanks for this advice and your time. $\endgroup$
    – Naveen Reddy Marthala
    Commented Nov 18, 2020 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.