I have a dataframe df_train of shape (11808, 1) that looks as follows:
Datum Menge
2018-01-01 00:00:00 19.5
2018-01-01 00:15:00 19.0
2018-01-01 00:30:00 19.5
2018-01-01 00:45:00 19.5
2018-01-01 01:00:00 21.0
2018-01-01 01:15:00 19.5
2018-01-01 01:30:00 20.0
2018-01-01 01:45:00 23.0
2018-01-01 02:00:00 20.5
2018-01-01 02:15:00 20.5
and a second df nan_df of shape (3071, 1) that looks as follows:
Datum Menge
2018-05-04 00:15:00 nan
2018-05-04 00:30:00 nan
2018-05-04 00:45:00 nan
2018-05-04 01:00:00 nan
2018-05-04 01:15:00 nan
2018-05-04 01:30:00 nan
2018-05-04 01:45:00 nan
2018-05-04 02:00:00 nan
2018-05-04 02:15:00 nan
The nan values in the nan_df need to be predicted using time series forecasting.
What I have done:
The code below divides the df df_train and runs the ARIMA model on that to predict the values for the test set
import pandas as pd
from pandas import datetime
import matplotlib.pyplot as plt
from statsmodels.tsa.arima_model import ARIMA
from sklearn.metrics import mean_squared_error
def parser(x):
return datetime.strptime(x,'%m/%d/%Y %H:%M')
df = pd.read_csv('time_series.csv',index_col = 1,parse_dates =[1], date_parser = parser)
df = df.drop(['Unnamed: 0'],axis=1)
df_train = df.dropna()
def StartARIMAForecasting(Actual, P, D, Q):
model = ARIMA(Actual, order=(P, D, Q))
model_fit = model.fit(disp=0)
prediction = model_fit.forecast()[0]
return prediction
NumberOfElements = len(df_train)
TrainingSize = int(NumberOfElements * 0.7)
TrainingData = df_train[0:TrainingSize]
TrainingData = TrainingData.values
TestData = df_train[TrainingSize:NumberOfElements]
TestData = TestData.values
#new arrays to store actual and predictions
Actual = [x for x in TrainingData]
Predictions = list()
#in a for loop, predict values using ARIMA model
for timepoint in range(len(TestData)):
ActualValue = TestData[timepoint]
Prediction = StartARIMAForecasting(Actual, 3, 1, 0)
print('Actual=%f, Predicted=%f' % (ActualValue, Prediction))
Predictions.append(Prediction)
Actual.append(ActualValue)
Error = mean_squared_error(TestData, Predictions)
print('Test Mean Squared Error (smaller the better fit): %.3f' % Error)
# plot
plt.plot(TestData)
plt.plot(Predictions, color='red')
plt.show()
Now, I wanted to do the same to predict the nan values in the nan_df, this time using the entire df_train dataframe and I did it as follows:
X = df_train.copy().values
nan_df = df.iloc[11809:, :].values
real = [x for x in X]
nan_Predictions = list()
#in a for loop, predict values using ARIMA model
for timepoint in range(len(nan_df)):
nan_ActualValue = nan_df[timepoint]
nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)
print('real=%f, Predicted=%f' % (nan_ActualValue, nan_Prediction))
nan_Predictions.append(nan_Prediction)
real.append(nan_ActualValue)
When I do this, I get the following error:
Traceback (most recent call last):
File "<ipython-input-42-33f3e242230d>", line 4, in <module>
nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)
File "<ipython-input-1-043dac0dd994>", line 17, in StartARIMAForecasting
model_fit = model.fit(disp=0)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 1157, in fit
callback, start_ar_lags, **kwargs)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 946, in fit
start_ar_lags)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 562, in _fit_start_params
start_params = self._fit_start_params_hr(order, start_ar_lags)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 539, in _fit_start_params_hr
if p and not np.all(np.abs(np.roots(np.r_[1, -start_params[k:k + p]]
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\lib\polynomial.py", line 245, in roots
roots = eigvals(A)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 1058, in eigvals
_assertFinite(a)
File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 218, in _assertFinite
raise LinAlgError("Array must not contain infs or NaNs")
LinAlgError: Array must not contain infs or NaNs
So, I would like to know how can I predict the nan values in the nan_df?
