0
$\begingroup$

I am trying to predict reservation count from a dataset with few features. Features are both categorical and continuous.

The dependent variable reservations looks like below: My dataset size is around 917 obs.

array([ 1,  7, 17,  2,  2, 13,  8, 11,  9,  4,  4,  3,  5,  2,  5,  7,  3,
       12,  9, 13,  5,  2, 11, 13, 14, 19,  9, 11,  3,  6,  7, 10,  1,  6,
        5, 10,  8,  5,  4,  3,  2, 10, 10, 10,  8, 13, 16,  6,  4,  6,  3,
       11, 10,  1, 18,  7,  2, 12, 17,  4,  2, 19,  3,  4, 17, 13, 10,  2,
       10,  1,  3,  4, 20,  3,  2,  1,  3,  5,  8,  8,  4,  3, 13,  3,  3,
        5,  4, 17,  7,  6, 10,  5,  3,  9,  9,  8,  1,  5, 17,  5, 10,  9,
        2,  7, 13,  2,  9,  1, 15, 13, 10,  4,  2,  4,  5,  4,  3,  3, 10,
        4,  7,  5, 13, 12,  7,  5,  6,  9,  5, 11,  7,  1,  4, 12,  4,  3,
       11,  1,  4,  4,  3,  7,  4, 11,  4,  1,  9,  2, 10, 10,  3,  4,  4,
        3,  2,  7, 10,  7,  6,  1,  3, 19,  9,  3,  8, 20,  1, 12,  9, 13,
       13,  2,  9,  4,  9,  2,  5,  6, 18,  3,  6,  8,  6,  4,  5, 13,  4,
        8,  9,  5,  4,  8,  5,  2,  1,  6,  8,  3,  6,  4,  2,  6, 11,  5,
        1,  5,  1,  5, 11, 11,  9,  3, 12,  2,  2,  9, 19,  7, 13, 13,  9,
        2,  1,  1,  4,  3,  4,  9,  1, 25, 12,  8,  5, 18,  3,  1,  6, 17,
        7,  4,  6,  9,  8, 10,  3,  8, 12,  5,  4,  4,  1,  9, 21,  4,  3,
        3,  7, 13,  5, 12,  8,  8,  6,  3,  6,  7,  5,  3,  7,  3, 14,  3,
        5,  2, 14, 16,  3,  8,  6, 13,  9,  3,  5,  4,  9,  4, 12, 12,  4,
        9,  8, 11,  5, 13,  3,  2,  5,  4,  2,  1,  8,  8, 18, 11,  2,  5,
       13,  4,  1,  2,  4,  1,  2,  2, 12,  2,  6, 19,  7, 20,  2, 10,  2,
        9, 12,  9,  8,  1,  4,  8,  8, 12,  4,  8,  1,  3,  6,  9,  4,  3,
        8,  2,  7, 15,  6,  5, 10,  6,  4,  3, 12,  5,  4, 13,  7,  2,  8,
        5,  2,  4,  3, 14, 12,  3,  4,  3,  2, 15,  6, 14, 12, 11,  9,  5,
        5,  7, 11, 10,  7,  9,  9,  7, 11,  5, 11,  3,  2,  5, 17,  5,  2,
        6,  1, 10,  3, 13, 19,  5,  1,  3,  5,  3,  5,  6,  3,  9,  8,  2,
        3,  2,  3,  7,  4,  9,  5,  1,  6, 14,  4,  8, 17, 13,  7,  1,  4,
        5, 10,  5,  6,  2, 12,  5,  9,  3,  9,  9,  1,  5,  1,  2,  2,  5,
        1,  4,  4, 13,  4, 25,  9, 10,  4,  3,  9, 13, 13,  2,  9,  2, 12,
        4,  1, 20,  9, 10,  2,  5,  4, 10,  2,  6,  1,  7,  7,  7,  4,  8,
        4,  3,  4, 13,  8,  3, 13, 12, 19,  9,  3,  2,  6,  7, 13,  8, 16,
        7,  3, 11,  4, 10,  9, 12,  2,  8,  5,  2,  3,  4,  2,  1, 11,  5,
        4,  2,  8, 12,  7,  5,  7,  7,  4,  6, 18,  2,  1,  6, 15, 11,  2,
        5,  8,  3,  5,  9, 11,  5,  8,  6, 20,  1, 10,  3,  7,  1,  3,  5,
        4,  4, 10, 11,  6,  1,  5,  4,  1,  2, 10,  4,  4, 11, 20,  5,  3,
        2,  7,  8,  2, 10,  5,  1, 18,  5, 10,  5,  3,  8, 15,  2,  1, 14,
       10,  7,  3,  5,  9,  3,  4, 21, 14,  1,  2,  1,  2,  4, 11,  9,  7,
        6,  9, 18,  4,  6, 18, 12, 12,  4,  6,  3,  3,  9,  5, 12, 15,  3,
        7,  3,  7,  4,  2, 15, 14,  7, 10,  5,  5,  5,  9,  3,  6,  3,  1,
       11,  1,  5, 25,  8,  2, 24,  1, 12,  1,  6,  8,  5, 13,  4,  3,  3,
       13,  4,  4, 18,  7, 13,  2,  8,  3,  4,  9,  2, 13, 12,  4,  5, 10,
        9, 15,  1,  8,  8, 15, 10,  1,  9,  2,  2,  2,  2,  3,  6, 17,  7,
        5,  5,  6, 12,  1,  8,  3,  1, 11,  4,  7,  8, 15,  6, 11,  9,  9,
       13,  2,  3,  5,  3,  5, 12,  4,  4,  8,  7, 12,  2,  2,  4,  4, 12,
        8, 11, 10,  6,  5,  1,  4,  2,  7,  3,  5, 15, 12, 12,  2,  9,  7,
        4,  4,  5, 15,  5,  8, 13,  7,  2,  8, 12,  2, 13,  6, 24, 14,  3,
        4,  1,  2,  8,  7,  5, 12,  8,  2,  6,  3,  7,  5,  2,  7,  3,  3,
        1,  9,  9,  3, 12,  3,  2, 11, 11,  6,  3,  9, 12,  4,  8,  7,  5,
        2, 10, 19,  1,  1, 10,  6,  2,  4,  2,  4,  4,  3,  7, 13,  9,  6,
        2,  2,  2,  5, 13, 12,  2, 13, 12, 11, 10,  5,  8,  8, 15, 12,  3,
        3,  9,  4,  6, 13, 15,  4,  7,  1, 12, 10,  9,  7,  3,  7,  4,  9,
        2, 10,  2, 11, 10, 14,  3, 13,  8,  3, 12, 11, 10,  7,  5,  3,  3,
       11,  3, 13,  9, 10, 20,  7, 12,  3,  6,  6, 18,  3, 10, 11, 10,  5,
        6, 11,  4,  6,  7,  9, 13,  1, 14, 14, 13,  4,  3,  8,  5,  7, 14,
       13, 13, 12,  8, 11, 12,  9,  8,  9,  4,  5,  4,  7,  5,  2,  3,  1,
        7,  2,  1, 13,  5, 19,  9,  6,  9,  7])

When I plot the histogram of dependent variable I get this

enter image description here

So I used a log transform to remove some of the skewness.

as y=np.log(df["reservartions"].values)

Now the plot of distribution looks below:

enter image description here

Some of features.

type    actual_price    recommended_price   num_videos  image_ava   text_length
1   67.85   59  5   0   7
0   100.70  53  5   0   224
0   74.00   74  4   1   21
0   135.00  75  1   0   184
0   59.36   53  2   1   31

Since actual_price and Recommended_price have huge correlation, I created a difference price of these two and dropped actual_price and recommended price.

But after running Linear regression or Random Forest Regression I get very poor results with R2 as 0.12 for both.

This shows the model is clearly not predicting and fitting well.

My dependent variable is clearly a positive variable. Is Linear Regression still right? Should I use Poisson regression? Log transformation makes sense?

EDIT:

Tried Poisson from Statsmodels. Gives worse results

    import statsmodels.api as sm 
    poisson_mod = sm.Poisson(train_Y, train_X)
    poisson_res = poisson_mod.fit(method="newton")
    print(poisson_res.summary())

Optimization terminated successfully.
         Current function value: 2.958960
         Iterations 5
                          Poisson Regression Results                          
==============================================================================
Dep. Variable:                      y   No. Observations:                  637
Model:                        Poisson   Df Residuals:                      628
Method:                           MLE   Df Model:                            8
Date:                Mon, 08 Oct 2018   Pseudo R-squ.:                 0.09479
Time:                        13:57:37   Log-Likelihood:                -1884.9
converged:                       True   LL-Null:                       -2082.2
                                        LLR p-value:                 2.506e-80
=================================================================================
                    coef    std err          z      P>|z|      [0.025      0.975]
---------------------------------------------------------------------------------
technology        0.0080      0.040      0.200      0.842      -0.070       0.086
street_parked     0.0014      0.030      0.046      0.963      -0.058       0.061
description       0.0002      0.000      0.884      0.377      -0.000       0.001
num_images_2     -0.0230      0.054     -0.430      0.667      -0.128       0.082
num_images_3      0.0619      0.053      1.160      0.246      -0.043       0.167
num_images_4      0.2234      0.050      4.501      0.000       0.126       0.321
num_images_5      0.2391      0.053      4.521      0.000       0.135       0.343
price_diff       -0.0146      0.001    -16.300      0.000      -0.016      -0.013
Bias              2.1325      0.052     41.016      0.000       2.031       2.234
=================================================================================
$\endgroup$
4
  • $\begingroup$ I recommend you these tasks, after that, you can put the result again: 1. remove outlier if exist 2. normalize variables 3. put price and percent of off furthermore, big bias means your model could not predict target. you should think about feature engineering $\endgroup$
    – parvij
    Oct 9, 2018 at 5:30
  • $\begingroup$ These are feature engg. What do u mean by normalizations? I have already standardized the continuous features $\endgroup$
    – Baktaawar
    Oct 10, 2018 at 17:22
  • $\begingroup$ data for linear regression should be Gaussian. Box-Cox transformation changes your data. try it. $\endgroup$
    – parvij
    Oct 11, 2018 at 5:25
  • $\begingroup$ That is not true. You can have features of any distribution. It doesn't need to be gaussian. Its the error which is gaussian $\endgroup$
    – Baktaawar
    Oct 11, 2018 at 23:00

3 Answers 3

1
$\begingroup$

One improvement would be a better representation of the categorical variables. These need to be represented differently before being included in the model.

For example, they could be one-HOT encoded https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html.

$\endgroup$
0
$\begingroup$

I think because your target is count so their distribution is Poisson and you should use Poisson regression.

I strongly recommend reading this, however, it's in R but will be helpful for you and also there is some similar python version in comments.

$\endgroup$
1
  • $\begingroup$ Pls check the edit of Poisson I tried $\endgroup$
    – Baktaawar
    Oct 8, 2018 at 21:20
0
$\begingroup$

I guess on the information provided a clear and specific answer can not be given. But some remarks and advice on the approach.

  • Log transformations can help if you face for example heteroscedasticity between your dependent variable and one of your independent variables (regressor).

  • Visualize / Explore your data set. What variables are strongly correlated with each other - Create for example a heatmap (e.g. in seaborn)

  • Start with a simple model. R² can be a misleading measurement and does not indicate if your model does perform well in prediction.

  • Based on your model output for the poisson regression it seems like your first five explanatory variables are not statistical significant - if your model was specfied correctly.

  • However, statistical significance does not imply that a variable is a good predictor or vice versa. Feature selection is widely discussed in research

  • Try several models and compare them by loss functions or gridsearchcv. Depending on your data set and data structure a Linear regression model may will do well. If you dealing with data that is highly non-linear and has a high demionsionality you may use NNs, Regression Trees etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.