I have a question about the type of model which I should use for a dataset I have.

The dataset has a total of 7 independent variables and 1 dependent variable which I need to predict.

Out of the 7 input variables, 6 of them are categorical and 1 is a date column. Now I have encoded the categorical columns using label encoding and converted them into numerical values. Now I’ve used a simple linear regression model on this dataset and achieved a normalized RMSE value of 0.11.

If I want to improve this accuracy, how do I go about doing it? How can I derive upon the kind of models that I can use considering the data set I have?

The data is mostly about forecast revenues for each product group. That's why I have those categorical columns which represent the hierarchy of the product groups.

  • $\begingroup$ What do you mean by 'hierarchy of the product groups'? $\endgroup$ – Elias Strehle Feb 11 '18 at 22:00
  • 1
    $\begingroup$ If we take a product like, say, a Honda Accord car, the "name" column will have "Accord", "make" column will have "Honda" , "class" will have "Sedan", "type" column will have "Four Wheeler" and so on. My columns will be something like "Type" -> "Class" -> "Make" -> "Name" Not exactly this but something similar. I hope you got the idea. $\endgroup$ – karun_r Feb 11 '18 at 23:34

I know you ask about the model choice here, but it is worth to discuss about your input data first. Data with many categorical features is still an active research; so it is not that straightforward. I suggest you first look at this very similar post, where I discuss some techniques to convert categorical variables to numerical values.

Since you provide little information about your categorical variables, for example how many levels each categorical variable have or how you do label encoding (just out-of-the-box method?) it is hard to give better guidelines. Generally speaking though, label encoding is mostly used for encoding target variables (dependent variable) if it is categorical, while you mention you use label encoding for other 6 independent variables (inputs). Here your target is rather numerical values (regression), so I would go beyond label encoding to convert your independent categorical variables.

Image the following very simplified scenario (just for educational purposes) to understand why label encoding is not a good idea where there is more than 1 independent categorical variables:

   Color   Year
0    Red   2010
1   Blue   2011
2  Green   2012

Now label encoding this would converting this to (naively speaking):

  Color  Year
0   1    1
1   2    2
2   3    3

There ways like constructing your own dictionary for each categorical variable and values associated to its levels while label encoding to prevent this obvious mistake. Apart from that, since you hardly go beyond RMSE value of 0.11, one could think of your input values (maybe not encoded properly!) and choice of your model.

Maybe try using this code for handling your independent categorical variables. Please note here, as a general rule of thumb, avoid using one-hot encoding although it is widely used in XGBoost (explained in the post). Maybe try "Target Encoding"!

Now speaking of the choice of model:

  1. Have you tried regularization (L1, L2)?
  2. Have confident you are about the independency of your independent variables? In simple regression you are assuming this holds, but may it is not which is easily the case that your independent variables are correlated. Maybe you should analyze this first using confusion matrix (see here for a brief tutorial)
  3. Have you tried gradient boosting decision trees (GBDT) for regression? Note1: If you want to go with GBDTs, do not leave that other 1 independent variable alone. You said you have 6 independent variables out of 7. I assume the other one is continuous numerical values? If that so, you have to handle that properly in GBDTs like binarizing (put in bins). Note2: You may try either XGBoost, or Catboost, and a nice tutorial using Catboost. Each has its pros and cons. For the former hyperparameter tuning could be challenging and there is no ways to automatically account for categorical variables but have larger community. While the latter is recently released (mid last year), it is much easier in terms of hyperparameter tuning (often defaults works great), and there are ways to automatically account for categorical features without explicitly encode them, but less support and community is smaller. Note3: The good thing about GBDT is that you do not need to worry about the correlation of features per se. Algorithm automatically ignore the coupling of strongly correlated features. Note4: You may easily overfit, although there are built-in regularization. Catboost for example have a interactive learning curve for training and validation to control overfitting (see the last tutorial).

I strongly believe taking care of your inputs together with a proper choice of algorithm would boost your RMSE error.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.