0
$\begingroup$

I have historical data from the MySQL DB which contains 33 months of data. The features in the data are state, depot, product type, purchase date, sales person name, volume, and price.

Using this information, I would like to predict/forecast product sales weekly, monthly, and yearly.

Since I only have 34 months of data, I do not think that I can apply statistical modeling techniques like Exponential Smoothing or ARIMA.

So, I just wanted to check if this data is sufficient for Machine learning modeling such as Linear Regression, SVM, or Random forest? Or any kind of feature engineering.

Note: I am unable to get more data, as DB has only 34 months of data.

Please suggest.

| improve this question | | | | |
$\endgroup$
0
$\begingroup$

There is no general answer to this question. Ethan's answer is correct in stating that more data is almost always desirable (up to the amount that your hardware can process quickly enough to meet your needs).

Less data means that you will be limited to fewer predictors (p >= n is typically a problem), and "noisier" results (point predictions will be less accurate, prediction ranges will be wider, classification accuracy will often be lower, etc.).

Machine learning can be done with a sample size of one-- the results will just probably not be worth very much. If that's all the data you have, that's all that you have. The question isn't so much can ML be done here? as will the output of a specific ML technique be good enough to use? Answering that question requires a look at your results and your needs more so than a rule of thumb about data quantity.

| improve this answer | | | | |
$\endgroup$
4
$\begingroup$

As @Ethan said there is no general answer to this question. There are multiple perspectives that you need to take in consideration.

  1. Amount of the data: The general tendency is more data will lead to better results.
  2. Variability of the data: A very important aspect is the variability of your data. If you have millions of data points that have almost the same input and output values it is empirically equivalent to a situation in which you only have one single data point.
  3. The timescale of data acquisition: The variability will strongly depend on the relevant timescale of the problem that you are trying to solve. If you are looking at products (e.g. tickets for public transportation) that are sold at a very high frequency the data acquisition rate should be shorter (short/fast time scale). If your data acquisition rate is too small your model will not be able to detect relevant changes e.g. tickets sales during the day. If you are looking at products (e.g. car sales) then your data acquisition rate should be longer (long/slow time scale). In the second case, I would still use the fastest possible data acquisition rate, because we can always reduce the given data. Note, that the data acquisition rate is linked to the variability of the data.
  4. Diversity of the data conditions: This is related to the variability of the data. But the focus here is on the conditions under which your data was collected. For example, imagine a situation in which we have some time series measurements of machines in a production plant during summer. If we know that the process is heavily dependent on the temperature we must also get measurements during spring, autumn, and winter in order to increase the diversity of our data set.

As the data problem is more involved it is difficult to give concrete numbers but the following rules of thumb can be applied. These must always be considered with the previous points.

  • In statistics, we often have the lower bound for samples to be $30$. This implies that you should never have less than $30$ observations.
  • For estimating parameters it is often advised to have $50$ (worst case) or better more than $200$ observations for each parameter that you want to estimate. For example, if you need to estimate $5$ parameters that would imply $250$ observations for the worst case and more than $1000$.
| improve this answer | | | | |
$\endgroup$
2
$\begingroup$

There is no minimum or maximum amount of data that is necessary to construct any Machine Learning model. The general rule, however, is that the more data that you add to the model, the better it will perform and generalize. Depending on the frequency of your data, 34 months of data should be sufficient for some basic exponential smoothing models or ARIMA as well.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ if I grouped all 34-month daily sales data on to the product type. Will ended up getting 3 data points for yearly prediction. Similarly 34 data points for monthly and 136 data points for weekly. So, you are saying 34 data points are enough for ESM and ARIMA modeling? $\endgroup$ – Optimizor May 8 '19 at 17:21
  • $\begingroup$ If your dataset contains 34 months of daily product sales, this would imply that you have around 30 days of observations for each month (this is plenty for a small model). While building a model with only 3 observations is technically possible, it will likely not perform well. $\endgroup$ – Ethan May 8 '19 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.