I am new to such panel data where I have multiple observation for same ID in different Quarter and I am not sure what kind of machine learning algorithm I can apply.

I have data from Q1-18 till Q4-2020

I have 2,000,000 rows and 200,000 unique id and 20 columns

For each id I have only 6-8 past quarter values, max quarter for each id are 8 quarters and for some id I have only 6 quarters where few quarter value are not available for that id

Below is the basic idea of what my data set look like

Quarter - respective business quarter for that year

Target - is the sales volume in ratio

I am trying to Predict - Target column for 2021 Q1 quarter

I have 8-10 different numeric columns and state , quarter and ID as category columns

I would appreciate if someone could suggest me what kind of modelling could be performed

enter image description here


Your data is a good candidate for a mixed effect model.

You have two potential random effects that look to be crossed: I see ID as one and state as another, as it seems that any ID can belong to any state.

Essentially, you can leverage the fact that some states and some IDs will have less data than others (and in general, some states and IDs will vary more than others). States and IDs who exhibit such characteristics will have their predictions bolstered by other states and IDs (called partial pooling). You also need to use time as a variable (perhaps convert quarter to 2019.25, 2019.5, etc. or use two time variables year, quarter) because you wish to predict the next period.

I think generalized linear mixed effect models can scale to 2 million observations. There are also machine learning methods that allow for mixed effects: mixed effect random forests (MixRF), mixed effect gradient boosting (GPBoost/mboost), and of course you can go full Bayesian as well (though you will likely need to use an approximation method since your data is large). The ML models may scale better.

  • $\begingroup$ Thanks for your input, what you are suggesting is to split in two variables 2019 and respective quarters, but If I training my model and the predict for 2021 then since the model is trained from 2018-2020 and when I have to predict for 2021 then while predicting I am not sure how will it work since 2021 time frame is not trained and only from 2019-2020 so when in producition data we will have 2021 and Q1 , How will the model predict since 2021 would be seen for 1st time. Not sure if I am able to make you understand what I meant $\endgroup$
    – Dexter1611
    Feb 19 at 20:04
  • $\begingroup$ I understand what you mean. Inevitably your model will have to extrapolate (i.e. learn the long term trend, if one exists) which may be a problem for some models that can't capture trends very well, like gradient boosting. If there are no trends then presumably behaviour from 2018, 2019, and 2020 are good indicators of 2021. I suggest figuring out first whether you target variable is trending over time, perhaps per state and/or per ID. If it is, you can account for this just like I said - include time as a predictor in a linear mixed effects model, and perhaps use polynomial or.... $\endgroup$
    – aranglol
    Feb 20 at 1:36
  • $\begingroup$ ...spline/smooth functions if it isn't linear. You can get pretty complex with this as well, where you can consider stuff like autocorrelation in the residuals if you are concerned that there is autocorrelation in time. $\endgroup$
    – aranglol
    Feb 20 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.