Right Regression Model to use

Question

I am trying to predict reservation count from a dataset with few features. Features are both categorical and continuous.

The dependent variable reservations looks like below: My dataset size is around 917 obs.

array([ 1,  7, 17,  2,  2, 13,  8, 11,  9,  4,  4,  3,  5,  2,  5,  7,  3,
       12,  9, 13,  5,  2, 11, 13, 14, 19,  9, 11,  3,  6,  7, 10,  1,  6,
        5, 10,  8,  5,  4,  3,  2, 10, 10, 10,  8, 13, 16,  6,  4,  6,  3,
       11, 10,  1, 18,  7,  2, 12, 17,  4,  2, 19,  3,  4, 17, 13, 10,  2,
       10,  1,  3,  4, 20,  3,  2,  1,  3,  5,  8,  8,  4,  3, 13,  3,  3,
        5,  4, 17,  7,  6, 10,  5,  3,  9,  9,  8,  1,  5, 17,  5, 10,  9,
        2,  7, 13,  2,  9,  1, 15, 13, 10,  4,  2,  4,  5,  4,  3,  3, 10,
        4,  7,  5, 13, 12,  7,  5,  6,  9,  5, 11,  7,  1,  4, 12,  4,  3,
       11,  1,  4,  4,  3,  7,  4, 11,  4,  1,  9,  2, 10, 10,  3,  4,  4,
        3,  2,  7, 10,  7,  6,  1,  3, 19,  9,  3,  8, 20,  1, 12,  9, 13,
       13,  2,  9,  4,  9,  2,  5,  6, 18,  3,  6,  8,  6,  4,  5, 13,  4,
        8,  9,  5,  4,  8,  5,  2,  1,  6,  8,  3,  6,  4,  2,  6, 11,  5,
        1,  5,  1,  5, 11, 11,  9,  3, 12,  2,  2,  9, 19,  7, 13, 13,  9,
        2,  1,  1,  4,  3,  4,  9,  1, 25, 12,  8,  5, 18,  3,  1,  6, 17,
        7,  4,  6,  9,  8, 10,  3,  8, 12,  5,  4,  4,  1,  9, 21,  4,  3,
        3,  7, 13,  5, 12,  8,  8,  6,  3,  6,  7,  5,  3,  7,  3, 14,  3,
        5,  2, 14, 16,  3,  8,  6, 13,  9,  3,  5,  4,  9,  4, 12, 12,  4,
        9,  8, 11,  5, 13,  3,  2,  5,  4,  2,  1,  8,  8, 18, 11,  2,  5,
       13,  4,  1,  2,  4,  1,  2,  2, 12,  2,  6, 19,  7, 20,  2, 10,  2,
        9, 12,  9,  8,  1,  4,  8,  8, 12,  4,  8,  1,  3,  6,  9,  4,  3,
        8,  2,  7, 15,  6,  5, 10,  6,  4,  3, 12,  5,  4, 13,  7,  2,  8,
        5,  2,  4,  3, 14, 12,  3,  4,  3,  2, 15,  6, 14, 12, 11,  9,  5,
        5,  7, 11, 10,  7,  9,  9,  7, 11,  5, 11,  3,  2,  5, 17,  5,  2,
        6,  1, 10,  3, 13, 19,  5,  1,  3,  5,  3,  5,  6,  3,  9,  8,  2,
        3,  2,  3,  7,  4,  9,  5,  1,  6, 14,  4,  8, 17, 13,  7,  1,  4,
        5, 10,  5,  6,  2, 12,  5,  9,  3,  9,  9,  1,  5,  1,  2,  2,  5,
        1,  4,  4, 13,  4, 25,  9, 10,  4,  3,  9, 13, 13,  2,  9,  2, 12,
        4,  1, 20,  9, 10,  2,  5,  4, 10,  2,  6,  1,  7,  7,  7,  4,  8,
        4,  3,  4, 13,  8,  3, 13, 12, 19,  9,  3,  2,  6,  7, 13,  8, 16,
        7,  3, 11,  4, 10,  9, 12,  2,  8,  5,  2,  3,  4,  2,  1, 11,  5,
        4,  2,  8, 12,  7,  5,  7,  7,  4,  6, 18,  2,  1,  6, 15, 11,  2,
        5,  8,  3,  5,  9, 11,  5,  8,  6, 20,  1, 10,  3,  7,  1,  3,  5,
        4,  4, 10, 11,  6,  1,  5,  4,  1,  2, 10,  4,  4, 11, 20,  5,  3,
        2,  7,  8,  2, 10,  5,  1, 18,  5, 10,  5,  3,  8, 15,  2,  1, 14,
       10,  7,  3,  5,  9,  3,  4, 21, 14,  1,  2,  1,  2,  4, 11,  9,  7,
        6,  9, 18,  4,  6, 18, 12, 12,  4,  6,  3,  3,  9,  5, 12, 15,  3,
        7,  3,  7,  4,  2, 15, 14,  7, 10,  5,  5,  5,  9,  3,  6,  3,  1,
       11,  1,  5, 25,  8,  2, 24,  1, 12,  1,  6,  8,  5, 13,  4,  3,  3,
       13,  4,  4, 18,  7, 13,  2,  8,  3,  4,  9,  2, 13, 12,  4,  5, 10,
        9, 15,  1,  8,  8, 15, 10,  1,  9,  2,  2,  2,  2,  3,  6, 17,  7,
        5,  5,  6, 12,  1,  8,  3,  1, 11,  4,  7,  8, 15,  6, 11,  9,  9,
       13,  2,  3,  5,  3,  5, 12,  4,  4,  8,  7, 12,  2,  2,  4,  4, 12,
        8, 11, 10,  6,  5,  1,  4,  2,  7,  3,  5, 15, 12, 12,  2,  9,  7,
        4,  4,  5, 15,  5,  8, 13,  7,  2,  8, 12,  2, 13,  6, 24, 14,  3,
        4,  1,  2,  8,  7,  5, 12,  8,  2,  6,  3,  7,  5,  2,  7,  3,  3,
        1,  9,  9,  3, 12,  3,  2, 11, 11,  6,  3,  9, 12,  4,  8,  7,  5,
        2, 10, 19,  1,  1, 10,  6,  2,  4,  2,  4,  4,  3,  7, 13,  9,  6,
        2,  2,  2,  5, 13, 12,  2, 13, 12, 11, 10,  5,  8,  8, 15, 12,  3,
        3,  9,  4,  6, 13, 15,  4,  7,  1, 12, 10,  9,  7,  3,  7,  4,  9,
        2, 10,  2, 11, 10, 14,  3, 13,  8,  3, 12, 11, 10,  7,  5,  3,  3,
       11,  3, 13,  9, 10, 20,  7, 12,  3,  6,  6, 18,  3, 10, 11, 10,  5,
        6, 11,  4,  6,  7,  9, 13,  1, 14, 14, 13,  4,  3,  8,  5,  7, 14,
       13, 13, 12,  8, 11, 12,  9,  8,  9,  4,  5,  4,  7,  5,  2,  3,  1,
        7,  2,  1, 13,  5, 19,  9,  6,  9,  7])

When I plot the histogram of dependent variable I get this

So I used a log transform to remove some of the skewness.

as y=np.log(df["reservartions"].values)

Now the plot of distribution looks below:

Some of features.

type    actual_price    recommended_price   num_videos  image_ava   text_length
1   67.85   59  5   0   7
0   100.70  53  5   0   224
0   74.00   74  4   1   21
0   135.00  75  1   0   184
0   59.36   53  2   1   31

Since actual_price and Recommended_price have huge correlation, I created a difference price of these two and dropped actual_price and recommended price.

But after running Linear regression or Random Forest Regression I get very poor results with R2 as 0.12 for both.

This shows the model is clearly not predicting and fitting well.

My dependent variable is clearly a positive variable. Is Linear Regression still right? Should I use Poisson regression? Log transformation makes sense?

EDIT:

Tried Poisson from Statsmodels. Gives worse results

    import statsmodels.api as sm 
    poisson_mod = sm.Poisson(train_Y, train_X)
    poisson_res = poisson_mod.fit(method="newton")
    print(poisson_res.summary())

Optimization terminated successfully.
         Current function value: 2.958960
         Iterations 5
                          Poisson Regression Results                          
==============================================================================
Dep. Variable:                      y   No. Observations:                  637
Model:                        Poisson   Df Residuals:                      628
Method:                           MLE   Df Model:                            8
Date:                Mon, 08 Oct 2018   Pseudo R-squ.:                 0.09479
Time:                        13:57:37   Log-Likelihood:                -1884.9
converged:                       True   LL-Null:                       -2082.2
                                        LLR p-value:                 2.506e-80
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
technology        0.0080      0.040      0.200      0.842      -0.070       0.086
street_parked     0.0014      0.030      0.046      0.963      -0.058       0.061
description       0.0002      0.000      0.884      0.377      -0.000       0.001
num_images_2     -0.0230      0.054     -0.430      0.667      -0.128       0.082
num_images_3      0.0619      0.053      1.160      0.246      -0.043       0.167
num_images_4      0.2234      0.050      4.501      0.000       0.126       0.321
num_images_5      0.2391      0.053      4.521      0.000       0.135       0.343
price_diff       -0.0146      0.001    -16.300      0.000      -0.016      -0.013
Bias              2.1325      0.052     41.016      0.000       2.031       2.234
=================================================================================

Answer 1

One improvement would be a better representation of the categorical variables. These need to be represented differently before being included in the model.

For example, they could be one-HOT encoded https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html.

Answer 2

I think because your target is count so their distribution is Poisson and you should use Poisson regression.

I strongly recommend reading this, however, it's in R but will be helpful for you and also there is some similar python version in comments.

Answer 3

I guess on the information provided a clear and specific answer can not be given. But some remarks and advice on the approach.

• Log transformations can help if you face for example heteroscedasticity between your dependent variable and one of your independent variables (regressor).

• Visualize / Explore your data set. What variables are strongly correlated with each other - Create for example a heatmap (e.g. in seaborn)

• Start with a simple model. R² can be a misleading measurement and does not indicate if your model does perform well in prediction.

• Based on your model output for the poisson regression it seems like your first five explanatory variables are not statistical significant - if your model was specfied correctly.

• However, statistical significance does not imply that a variable is a good predictor or vice versa. Feature selection is widely discussed in research

• Try several models and compare them by loss functions or gridsearchcv. Depending on your data set and data structure a Linear regression model may will do well. If you dealing with data that is highly non-linear and has a high demionsionality you may use NNs, Regression Trees etc.