I have tried to build a model to forecast the count of a particular variable.The model that was used for the purpose was poisson .Unfortunately ,i don't have enough stat knowledge to analyze the model performance .If somebody can provide some insights as of how the model is performing,as well as some tweaks to improve the model performance will be greatly helpful. I am also willing to try out other models if it performs better.

I am using python with the statmodels package to build the model.

Attaching a graph which shows the fitted and the actual values(Green shows the actual values and Blue shows the fitted values)

enter image description here Also,providing the summary() output of the model

                 Generalized Linear Model Regression Results                  
Dep. Variable:         Work_Item_Type   No. Observations:                  581
Model:                            GLM   Df Residuals:                      574
Model Family:                 Poisson   Df Model:                            6
Link Function:                    log   Scale:                             1.0
Method:                          IRLS   Log-Likelihood:                -16752.
Date:                Mon, 22 Feb 2016   Deviance:                       31268.
Time:                        21:59:12   Pearson chi2:                 1.05e+05
No. Iterations:                     9                                         
                  coef    std err          z      P>|z|      [95.0% Conf. Int.]
Intercept       2.8492      0.051     55.426      0.000         2.748     2.950
Weekday        -0.2066      0.032     -6.446      0.000        -0.269    -0.144
day_of_week    -0.0926      0.007    -13.367      0.000        -0.106    -0.079
wom             0.1122      0.007     16.996      0.000         0.099     0.125
week           -0.0411      0.001    -53.597      0.000        -0.043    -0.040
TimeDelta       0.0001    5.1e-05      2.933      0.003      4.96e-05     0.000
month_of_yr     0.2192      0.004     60.981      0.000         0.212     0.226

Also attaching a sample of the dataset used 

clear_date  Count_Work_Item_Type
7/7/2014    1
7/10/2014   1
7/11/2014   5
7/17/2014   2
7/22/2014   1
7/24/2014   1
7/29/2014   3
7/30/2014   4
8/13/2014   1

Since i had only the date and the variable to be forecast i created a bunch of other variables like

Day of week
Week of Month
Time Delta (Starts from 0 increment by one until end)
Month of Year

Also,i haven't done any kind of transformation on the variables. Please do comment if you need additional information: Thhanks

  • $\begingroup$ How did you create your variables? Is it treating your categorical variables (day of week, for example) as a multi-level factor or as a numeric linear term? The fact you only get one coefficient for day_of_week makes me think the latter. Do you really expect the count to be linear during a week? You should probably do some basic reading of GLMs... $\endgroup$ – Spacedman Feb 22 '16 at 22:52

I'm not sure what you mean by "performance", but if what you mean is fit the answer is clear. You need to be using the log-likelihood to differentiate between different models. Basically, when you are fitting the model you are trying to maximize the log-likelihood. Thus the log-likelihood is giving you some sense of how well the parameters of your model are doing at fitting the data.

In your case, you want to get log-likelihood to be as close to zero as possible. Now this is kind of terrible advice, because if you were clever enough to come up with a feature for every observation, you could get a perfect fit. That's bad because your model would be completely useless. There are functions that take your log-likelihood as an input and transform it to penalize you for adding more variables, etc. We won't worry about those right now. Just keep in mind not to mindlessly lower the log-likelihood.

Once you have a model that you can live with you should run some sort of cross-validation, and/or hold out set. Then you can use any number of metrics to validate the predictive performance of your model. I think that is the more important of the two issues. You could calculate the mean square error on your hold out set. $$MSE=\sum_{i=1}^n(\hat y_i - y_i)^2$$ This would give you a really basic metric to assess how well your model is predictive of the output.

| improve this answer | |


If you want more exposure to time series forecasting specifically with count data I would recommend these slides.

Main things to remember:

  • Check your data first. To use any model you generally have some assumptions that the model imposes on you. Plot your response variable, your time series, etc.
  • Count models are useful for skewed data (that isnt normal given some transform: log, power, boxcox) when normal models arent appropriate.
  • Check if the your timeseries data meets the assumptions of the poisson model. For example, that the variability is equal to the mean. Check for equidisperion.
  • There are several other tests you should check before modeling: Correlograms, Augmented Dickey Fuller Test, PACF Plots, ACF Plots

Do the main point of your question: How is my model performing?

  • Standard performance metrics: MAE, MAPE, MSE

Given that you may not have a bunch of exposure to timeseries modeling, maybe check out Facebook's Prophet package. It was specifically designed for data analysts who dont need/want to worry too much about model selection.

That being said, prophet also has a lot of tuning parameters and things. At the very least make sure you research the assumptions of these models before you use them!

| improve this answer | |
  • $\begingroup$ Hi, welcome to DS. Generally, link-only answers are not so well received here in the forum. Please consider to at least add a summary of important aspects. Cheers... $\endgroup$ – Peter Jan 2 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.