# Sparse Data Interpolation

I have a price data set where on some days there are up to five data points and some days none at all. For example:

2.110   2017-04-19
1.910   2017-04-23
1.980   2017-04-24
1.980   2017-04-24
1.980   2017-04-24
1.230   2017-04-24
2.100   2017-05-04
1.920   2017-05-08


The total data set is around 500 points.

What is the best way to go about interpolating this data and making it's frequency daily?

I'm considering using Gaussian Process Modelling using PyMC3 but would really appreciate any ideas or advice. Additionally, the volatility of the price data must be preserved as best as possible as this will be input to a volatility model later on.

If you know that the multiple prices on a single day are chronological (and you don't have the actual timestamps), I would suggest simply taking the last price. This is then as close as you can get to using the Closing Price, which is the most common. Often you have Open, High, Low, Close, for example, but just use the Close prices.

Interpolating, or imputing, the data can be done in many ways. One factor that might help decide on a method, is how many of the points are missing? You have ~500 data points: how many days in total? how many of those require imputing?

For a relatively low number of missing values, scattered more or less randomly over your time-series, I would suggest going for simpler imputation methods, rather than model-based one.

A model-based approach can introduce other biases/problems that would potentially be harder to debug and more difficult to understand, so I would hold off there initially.

### Forward-filling

This is my go to method, to just get started. If only a few percent of the entire time-series are missing, something as simple as fill-forward might be acceptable. Even the gaps in data are randomly or evenly spaced along the timeline, this is also a reasonable approach.

A nice property of this method is that it agrees with the theory on stochastic price-paths. These are usually modelled as Gaussian random variables, following the principles or Brownian motion. This all culminates in the idea of a Martingale Process, which states:

at a particular time in the realized sequence, the expectation of the next value in the sequence is equal to the present observed value even given knowledge of all prior observed values.

### Random sampling

One way to fill a seemingly stochastic price-path, would be to compute the mean and variance of the dataset, then draw random samples from that distribution to fill missing values. This should preserve the mean and variance of the dataset's distribution quite well (and fairly).

Here is is important to realise that you may inadvertently be inducing a bias within that dataset, namely that you have use population statistics to alter the data. This means, depending on your model testing method later on, that you violate information flow through time. E.g. when predicting the 100th step, using the 99 preceding steps, it could be the case that the 99th step was one you imputed. Using the mean and variance of the entire dataset to do so has inherently encoded some information from the 100th step (and all following steps) into that 99th step. This is a subtle detail, but something to be aware of.

If you can put the time-series in a Pandas DataFrame, it might give you some more ideas by skimming the DataFrame's built-in methods to handle missing data.

Here is a more in-depth analysis of possibilities with some good explanations.