1
$\begingroup$

I have below datasets for two years each holding about 10.000 records. Every week a new report is generated that shows the performance for the current or any previous month. Therefore a more recent weekly report would overwrite the monthly value of a previous report (for that specific month). I need to predict this year's performance based on the weekly data coming in.

structure of a full year data set

I have two questions:

  1. How would I need to structure my dataset for machine learning? If I keep the above I would get about 150 features (3 per week) and I do not know whether there is even a ML algorithm that can handle the many n/a (all forthcoming weeks) for the year for which I need to run the prediction. I can also transform this into a new dataset by combining the three columns by week into three features (would result in long narrow table). But that would replicate each entity and output value about 50 times (once per week).
  2. Based on the answer to 1. what would be suitable training algorithms?
Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I guess you need to predict following week's performance based on previous month's performance. Well, I think its more or less like predicting tomorrow's weather based on previous data.

I guess you need to use Time series forecasting model (ARIMA or LSTM ) for the same by setting the lag for one month. You can build a simple Deep Learning model for the same by adding layers with lag as a parameter

Improve this answer
$\endgroup$
2
  • $\begingroup$ Few questions on this approach: How do I exploit that from previous years I know the expected outcome at the end of the time series presumably also potentially depending on the categorical features? Would I implement ARIMA with a one year seasonality because every year I start from 0? Not sure also whether I can set the lag to one month since any current or previous month may be reported (overwrites previous values) in a weekly report. $\endgroup$
    – Bernd
    Jan 8, 2019 at 8:53
  • $\begingroup$ You can implement ARIMA with one year seasonality. I was mentioning about Recurrent NN where the model learns from the previous inputs $\endgroup$
    – Sunil
    Jan 15, 2019 at 9:15
0
$\begingroup$

It looks like you receive the same 4 features every week - i.e. ID, calendar week, calendar month, monthly performance.

I would recommend stacking vertically - adding each new week's data to the bottom of the existing data.

It may also be a good idea to create additional features to improve prediction. Some examples may include creating some form of date variable or some rolling annual performance total. Some algorithms allow more emphasis to be placed on certain observations; this can be handy in time-series problems as you can place more weight on more recent data.

Finally, you may need to perform some data transformation - e.g. encoding categorical variables, scaling / centring numerical variables. This really is dependent on the algorithm you choose to use and may be unnecessary in some cases.

Improve this answer
$\endgroup$
1
  • $\begingroup$ In this set-up I would have each entity (ID; not used for the model) including categories and annual performance (output variable) repeated 50 times in my training dataset. Also from the weekly report each entity would be repetitively added to the dataset. Would that cause any issue for training or prediction? $\endgroup$
    – Bernd
    Jan 8, 2019 at 8:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.