I have a dataset like this, basically all numerical time-series data.

enter image description here

I would like to generate dummy/artificial/fake data for future values of this, preferably in python. How can I achieve this for reasonable looking future values?

I checked some packages like faker, Timesynth; but couldn't figure out how to produce data which considers previous data.

  • $\begingroup$ What do you mean by generating "reasonable" dummy variables? $\endgroup$ – Ethan Dec 7 '20 at 16:18
  • $\begingroup$ @Ethan I would like to generate data from the potential underlying distribution of each column without discarding their correlation to each other. Another question would be let's say I found future data for column 2, can I generate reasonable data (meaning previous correlations are not discarded) for the future values of the rest of the columns. $\endgroup$ – dafdaf Dec 7 '20 at 22:20
  • $\begingroup$ @Ethan 1)Another way to put it would be, how to forecast data without losing correlation of variables, 2) if you have values for one of the variables (dependent one) can you forecast the rest of the variables. $\endgroup$ – dafdaf Dec 7 '20 at 22:48

This problem is referred to as scenario generation, commonly used in stochastic optimization. The key problem is that you have to respect correlations among the different variables, which means that you need to model the joint multivariate predictive distribution. One way to bypass this is to use Copulas, which only require modelling the marginal probability densities. An approach widely used in engineering is the following:

  1. Model the marginal (empirical) probability distribution of each variable separately. You can also model the conditional probability distribution using probabilistic forecasting if needed.
  2. Estimate the empirical correlation matrix between the time series.
  3. Use the empirical correlation matrix and the marginal conditional distributions to fit a multivariate Copula.
  4. Obtain correlated scenarios by sampling from the multivariate Copula.

I am including a standard reference from the field of wind forecasting, which models the temporal correlation from the same time series. This can be applied to the case of different time series (spatial correlation).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.