I'm working on my first time series project where I am required to produce predictions for financial data.
The raw data is below:
Clearly, there is a seasonality and downward trend, I used the following code to normalize, and remove the seasonality, trend and volatility:
#Normalize
avg, dev = df.mean(), df.std()
df = (df - avg) / dev
plt.plot(df)
plt.show()
#Diff to remove trend
df = df.diff().dropna()
plt.plot(df)
plt.show()
# Remove Volatility with std.
ann_vol = df.groupby(df.index.year).std()
ann_vol = df.index.map(lambda d: ann_vol.loc[d.year])
ann_vol = np.stack(ann_vol).astype(None)
df = df / ann_vol
mon_vol = df.groupby(df.index.month).mean()
mon_vol = df.index.map(lambda d: mon_vol.loc[d.month])
mon_vol = np.stack(mon_vol).astype(None)
df = df - mon_vol
From here, I ran an adfuller test with the following result:
ADF Test Statistic : -3.021531399549568
p-value : 0.03293197916862391
#Lags Used : 1
Number of Observations Used : 48
P value is less than 0.05 that means we can reject the null hypothesis(Ho).
Therefore we can conclude that data has no unit root and is stationary
At this point, I ran auto_arima for the best p,q,d values:
model = pm.auto_arima(train_data,
m=12,
seasonal=True,
start_p=0,
start_q=0,
#D=1,
max_order=48,
test='adf',
error_action='ignore',
stepwise=True,
trace=True)
model.summary()
Result of auto_arima:
Best model: ARIMA(4,2,0)(0,0,0)[12]
Total fit time: 3.322 seconds
I split the train and test splits on dates, leaving the last 3 months of data in the test set and ran the model:
SARIMAX Results
==============================================================================
Dep. Variable: Volume No. Observations: 47
Model: SARIMAX(4, 2, 0) Log Likelihood -62.195
Date: Mon, 13 Feb 2023 AIC 134.390
Time: 00:43:01 BIC 143.423
Sample: 11-30-2018 HQIC 137.757
- 09-30-2022
Covariance Type: opg
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
ar.L1 -1.2511 0.167 -7.482 0.000 -1.579 -0.923
ar.L2 -0.8985 0.311 -2.892 0.004 -1.507 -0.290
ar.L3 -0.6493 0.311 -2.089 0.037 -1.258 -0.040
ar.L4 -0.2662 0.228 -1.167 0.243 -0.713 0.181
sigma2 0.8921 0.274 3.254 0.001 0.355 1.430
===================================================================================
Ljung-Box (L1) (Q): 0.00 Jarque-Bera (JB): 1.39
Prob(Q): 0.98 Prob(JB): 0.50
Heteroskedasticity (H): 1.81 Skew: 0.13
Prob(H) (two-sided): 0.26 Kurtosis: 2.18
===================================================================================
Warnings:
[1] Covariance matrix calculated using the outer product of gradients (complex-step).
Results were as follows, on the test data:
Factor Actual Volume Pred Volume
2022-10-31 0.250243 -0.674529
2022-11-30 0.939307 -0.781658
2022-12-31 0.156945 -1.370364
Clearly not a good result, but more interesting is that my residual table had no values:
2022-10-31 , 2022-11-30 , 2022-12-31 , Volume
2022-10-31 NaN NaN NaN NaN
2022-11-30 NaN NaN NaN NaN
2022-12-31 NaN NaN NaN NaN
Is this normal behavior? The lack of residual values implies that maybe I did something wrong or missed something? I think that maybe using a univariate time series to make predictions on this type of data isn't the correct route and maybe a multivariate approach would be more useful?
To confirm my suspicions, I ran the raw data through Prophet and got, well, this:
ds yhat yhat_lower yhat_upper
50 2022-12-31 19381.941721 17270.308719 21596.883702
51 2023-01-31 20515.256875 18104.569049 22721.360633
52 2023-02-01 15350.602454 13180.210930 17887.406605
53 2023-02-02 9990.535053 7847.143896 12063.906285
54 2023-02-03 4546.140197 2504.063878 6761.034962
