3
$\begingroup$

I have a dataframe df_train of shape (11808, 1) that looks as follows:

Datum   Menge
2018-01-01 00:00:00 19.5
2018-01-01 00:15:00 19.0
2018-01-01 00:30:00 19.5
2018-01-01 00:45:00 19.5
2018-01-01 01:00:00 21.0
2018-01-01 01:15:00 19.5
2018-01-01 01:30:00 20.0
2018-01-01 01:45:00 23.0
2018-01-01 02:00:00 20.5
2018-01-01 02:15:00 20.5

and a second df nan_df of shape (3071, 1) that looks as follows:

Datum                    Menge
2018-05-04 00:15:00       nan
2018-05-04 00:30:00       nan
2018-05-04 00:45:00       nan
2018-05-04 01:00:00       nan
2018-05-04 01:15:00       nan
2018-05-04 01:30:00       nan
2018-05-04 01:45:00       nan
2018-05-04 02:00:00       nan
2018-05-04 02:15:00       nan

The nan values in the nan_df need to be predicted using time series forecasting.

What I have done:

The code below divides the df df_train and runs the ARIMA model on that to predict the values for the test set

import pandas as pd
from pandas import datetime
import matplotlib.pyplot as plt
from statsmodels.tsa.arima_model import ARIMA
from sklearn.metrics import mean_squared_error

def parser(x):
    return datetime.strptime(x,'%m/%d/%Y %H:%M')

df = pd.read_csv('time_series.csv',index_col = 1,parse_dates =[1], date_parser = parser)
df = df.drop(['Unnamed: 0'],axis=1)
df_train = df.dropna()

def StartARIMAForecasting(Actual, P, D, Q):
    model = ARIMA(Actual, order=(P, D, Q))
    model_fit = model.fit(disp=0)
    prediction = model_fit.forecast()[0]
    return prediction

NumberOfElements = len(df_train)

TrainingSize = int(NumberOfElements * 0.7)
TrainingData = df_train[0:TrainingSize]
TrainingData = TrainingData.values
TestData = df_train[TrainingSize:NumberOfElements]
TestData = TestData.values

#new arrays to store actual and predictions
Actual = [x for x in TrainingData]
Predictions = list()

#in a for loop, predict values using ARIMA model
for timepoint in range(len(TestData)):
    ActualValue = TestData[timepoint]
    Prediction = StartARIMAForecasting(Actual, 3, 1, 0)
    print('Actual=%f, Predicted=%f' % (ActualValue, Prediction))
    Predictions.append(Prediction)
    Actual.append(ActualValue)

Error = mean_squared_error(TestData, Predictions)
print('Test Mean Squared Error (smaller the better fit): %.3f' % Error)
# plot
plt.plot(TestData)
plt.plot(Predictions, color='red')
plt.show()

Now, I wanted to do the same to predict the nan values in the nan_df, this time using the entire df_train dataframe and I did it as follows:

X = df_train.copy().values
nan_df = df.iloc[11809:, :].values

real = [x for x in X]
nan_Predictions = list()

#in a for loop, predict values using ARIMA model
for timepoint in range(len(nan_df)):
    nan_ActualValue = nan_df[timepoint]
    nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)
    print('real=%f, Predicted=%f' % (nan_ActualValue, nan_Prediction))
    nan_Predictions.append(nan_Prediction)
    real.append(nan_ActualValue)

When I do this, I get the following error:

Traceback (most recent call last):

  File "<ipython-input-42-33f3e242230d>", line 4, in <module>
    nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)

  File "<ipython-input-1-043dac0dd994>", line 17, in StartARIMAForecasting
    model_fit = model.fit(disp=0)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 1157, in fit
    callback, start_ar_lags, **kwargs)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 946, in fit
    start_ar_lags)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 562, in _fit_start_params
    start_params = self._fit_start_params_hr(order, start_ar_lags)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 539, in _fit_start_params_hr
    if p and not np.all(np.abs(np.roots(np.r_[1, -start_params[k:k + p]]

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\lib\polynomial.py", line 245, in roots
    roots = eigvals(A)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 1058, in eigvals
    _assertFinite(a)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 218, in _assertFinite
    raise LinAlgError("Array must not contain infs or NaNs")

LinAlgError: Array must not contain infs or NaNs

So, I would like to know how can I predict the nan values in the nan_df?

$\endgroup$
1
$\begingroup$

You may apply Wolfram Language to your project. There is a free Wolfram Engine for developers and with the Wolfram Client Library for Python you can use these functions in Python.

I will first create some data (too few rows provided in OP) in a pandas.DataFrame using a Python WolframLanguageSession to simulate a ARIMAProcess with RandomFunction.

Imports

import pandas as pd
import iso8601

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

Start WolframLanguageSession

wolfSession = WolframLanguageSession();

A simulation can be run by

print(
    wolfSession.evaluate(
        wl.RandomFunction(
            wl.ARIMAProcess(1.4, [-0.9], 1, [-0.01, -0.08], 2.6), 
            [0, 3]
        )('Values')
    )
)
[1.1054178694529107, 1.860340990531042, 1.5519448249118848, 5.088452598965132]

Run a simulation of 100 steps (n) with dates in 15 minute intervals as in example data snip.

n=100;
df = pd.DataFrame(
        {
        'Datum' : pd.date_range('2018-01-01 00:00', periods=n, freq=pd.offsets.Minute(15)),
        'Menge' : wolfSession.evaluate(
                    wl.RandomFunction(
                        wl.ARIMAProcess(1.4, [-0.9], 1, [-0.01, -0.08], 2.6), 
                        [0, n-1]
                    )('Values')
                )
        },
    );

The result can be visualised with DateListPlot by Exporting in one of the supported Raster Image Formats or Vector Graphics Formats.

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>', 
        wl.DateListPlot(wl.Query(wl.Values,wl.Values)(df), PlotTheme='Detailed')
    )
)

Mathematica graphics

Now that there is some data a forecast can be performed. First TimeSeriesModelFit will be use to fit an ARIMA process. Then TimeSeriesForecast to predict future values.

The Wolfram Engine interprets a pandas.DataFrame as a Dataset object. TimeSeriesModelFit expects a TimeSeries or a list of time-value pairs. Therefore, using Query the conversion is made to a list of time-value pairs.

ts_model=wolfSession.evaluate(
    wl.Query(
        wl.RightComposition(
            wl.Values,
            wl.Function(wl.TimeSeriesModelFit(wl.Slot(1),'ARIMA'))
        ),
        wl.Values
    )(df)
);

print(wolfSession.evaluate(ts_model('BestFit')))
ARIMAProcess[1.4524650139005593, [-0.9099324923212446], << 1 >>, [-0.07171874225371022], 2.507600357444524]

20 steps forward can be simulated with

ts_forecast=wolfSession.evaluate(wl.TimeSeriesForecast(ts_model,[20]));

print(wolfSession.evaluate(ts_forecast('Values')))
[69.49895300256293, 73.9505213906962, 71.35235968644419, 75.16897645534839, 73.14857786048489, 76.43946920329194, 74.89744525567367, 77.75304796344956, 76.60710728838431, 79.10230095679928, 78.28430817717482, 80.48109139973985, 79.93463198084233, 81.88433817573274, 81.56270217242249, 83.30783423643538, 83.17234688189897, 84.74809624199086, 84.76673571338941, 86.20224006662474]

ts_forecast is a TemporalData object whose properties include 'Dates' and 'Values'. These can be use to convert it into a Python pandas.DataFrame for further processing in Python.

df_forecast = pd.DataFrame(
        {
        'Datum' : list(map(
                    lambda d: iso8601.parse_date(d), 
                    wolfSession.evaluate(
                        wl.Map(
                            wl.Function(wl.DateString(wl.Slot(1),'ISODateTime')),
                            ts_forecast('Dates')
                        )
                    )
                )),
        'Menge' : wolfSession.evaluate(ts_forecast('Values'))
        },
    );

print(df_forecast.iloc[:3,:])
                      Datum      Menge
0 2018-01-02 01:00:00+00:00  69.498953
1 2018-01-02 01:15:00+00:00  73.950521
2 2018-01-02 01:30:00+00:00  71.352360

Further processing can also be continued with the Wolfram Engine. For example, 95% confidence interval bands for the forecast.

conf = .95;
quant = wolfSession.evaluate(wl.Quantile(wl.NormalDistribution(), 1 - (1 - conf) / 2));
errors = wolfSession.evaluate(wl.Sqrt(ts_forecast('MeanSquaredErrors')));
error_bands = wolfSession.evaluate(
                wl.TimeSeriesThread(
                    wl.Function([wl.Dot([1, -quant], wl.Slot(1)), wl.Dot([1, quant], wl.Slot(1))]),
                    [ts_forecast, errors]
                )
            );

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>', 
        wl.DateListPlot(
            [wl.Query(wl.Values,wl.Values)(df), error_bands, ts_forecast], 
            PlotStyle=[wl.Automatic, wl.Gray, wl.Automatic], 
            Filling=[wl.Rule(2,[3])],
            PlotTheme='Detailed'
        ) 
    )
);

Mathematica graphics

Terminate the session

wolfSession.terminate();

Hope this helps.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.