4
$\begingroup$

I have some time series, like this one: hash_rate I want to predict future values, so I splitted in train/test (70/30) and I created several ARIMA models, however they are all completely wrong (or maybe I am wrong). First of all, considering differentiation, ACF and PACF, I supposed that a good model can be an ARIMA(2,2,2) or something similar (spoiler: I did not found any good model) acf_pacf I transformed the data with a box-cox transformation, in order to have more similar values, so from

            Timestamp   Value           year    month
Timestamp               
2010-01-01  2010-01-01  5.948447e+06    2010    Jan
2010-01-01  2010-01-01  1.116364e+07    2010    Jan
2010-01-02  2010-01-02  5.272979e+06    2010    Jan
2010-01-02  2010-01-02  1.015796e+07    2010    Jan
2010-01-03  2010-01-03  1.091864e+07    2010    Jan

to

            Timestamp   Value       year    month
Timestamp               
2010-01-01  2010-01-01  29.411458   2010    Jan
2010-01-01  2010-01-01  31.462963   2010    Jan
2010-01-02  2010-01-02  29.029479   2010    Jan
2010-01-02  2010-01-02  31.149175   2010    Jan
2010-01-03  2010-01-03  31.389007   2010    Jan

As input target variable I stated "Value". Then, I found this nice function that allows to create SARIMAX models:

def model_auto_sarimax(y, seasonality, seasonal_flag, exogenous_variable):
    
    # Train model
    model = pm.auto_arima(train[input_target_variable], exogenous=exogenous_variable, 
                          start_p = 1, start_q = 1, 
                          max_p = 3, max_q = 3, m = input_seasonality, 
                          start_P = 0, seasonal = seasonal_flag, 
                          d = None, max_D = 1, trace = True, 
                          error_action ='ignore',   
                          suppress_warnings = True,  stepwise = True, 
                          max_order=12)
    
    # Model summary 
    print(model.summary())
    
    # Model diagnostics
    model.plot_diagnostics(figsize=(10,7))
    plt.show()
    
    return model

And finally, this function to get predictions:

def get_predictions(input_ts_algo, model, train, test, input_target_variable, exogenous_variable = None):
    
    print("------------- Get Predictions --------------- \n")
    # Get prediction for test duration
    if input_ts_algo == "manual_sarima":
        predictions = pd.Series(model.predict(len(train) + 1, len(train) + len(test), typ = 'levels').rename("Predictions")).reset_index(drop = True)
    elif input_ts_algo in ["auto_arima", "auto_sarima", "auto_sarimax"]:
        predictions = pd.Series(model.predict(len(test), exogenous = exogenous_variable)).reset_index(drop = True)
    else:
        predictions = pd.Series(model.forecast(len(test))).reset_index(drop = True)
    return predictions

Now, considering a SARIMA model I call the previous declared functions (and some others)

    print("------------- Auto SARIMA --------------- \n")
    model = model_auto_sarimax(y = train[input_target_variable], seasonality = 7, seasonal_flag = True, exogenous_variable = None)
    predictions = get_predictions(input_ts_algo, model, train, test, input_target_variable, exogenous_variable = None)
    evaluate_model(actuals, predictions)

(Please note that seasonality=7 is only an attempt, because I tried with lot of values, but I am not sure of which using, this is only an example). These are some results:

------------- Auto SARIMA --------------- 

Performing stepwise search to minimize aic
 ARIMA(1,1,1)(0,0,1)[7] intercept   : AIC=12761.725, Time=0.92 sec
 ARIMA(0,1,0)(0,0,0)[7] intercept   : AIC=14108.718, Time=0.06 sec
 ARIMA(1,1,0)(1,0,0)[7] intercept   : AIC=13227.472, Time=0.45 sec
 ARIMA(0,1,1)(0,0,1)[7] intercept   : AIC=12759.944, Time=0.69 sec
 ARIMA(0,1,0)(0,0,0)[7]             : AIC=14113.178, Time=0.06 sec
 ARIMA(0,1,1)(0,0,0)[7] intercept   : AIC=12757.961, Time=0.31 sec
 ARIMA(0,1,1)(1,0,0)[7] intercept   : AIC=12759.942, Time=0.56 sec
 ARIMA(0,1,1)(1,0,1)[7] intercept   : AIC=12760.164, Time=1.77 sec
 ARIMA(1,1,1)(0,0,0)[7] intercept   : AIC=12759.758, Time=0.45 sec
 ARIMA(0,1,2)(0,0,0)[7] intercept   : AIC=12759.784, Time=0.53 sec
 ARIMA(1,1,0)(0,0,0)[7] intercept   : AIC=13226.206, Time=0.16 sec
 ARIMA(1,1,2)(0,0,0)[7] intercept   : AIC=12759.991, Time=1.36 sec
 ARIMA(0,1,1)(0,0,0)[7]             : AIC=12869.046, Time=0.10 sec

Best model:  ARIMA(0,1,1)(0,0,0)[7] intercept
------------- Get Predictions --------------- 

------------- Model Evaluations --------------- 

MAPE :  10.758296890064576
MAE  :  43.21650273949536
RMSE  :  51.58017099571381
R2 Score  :  -10.690625135150933
Durbin Watson Score :  0.005276384675653324

predictions As you can see predictions are completely different. Model is predicting a straight line. Obviously also after anti-transforming the values after normalization: results So, I am very confused. I do not know all these models are so poor. I tried all possible configurations, changing models, seasonality and so on. Moreover, I tried also with a smaller series, removing data before 2018 because they are very small, but also in this case I have bad results. My question is: there is something strange in the code, or I am wrong with the models and parameters?

EDIT: This is how I import my data

hash_rate = pd.read_csv("data/bitcoin-mean-hash-rate.csv")

hash_rate["Hash Rate/t"] = hash_rate["Hash Rate/t"].str.rstrip("T00:00:00.000Z")
hash_rate["Hash Rate/t"] = pd.to_datetime(hash_rate["Hash Rate/t"])
hash_rate = hash_rate.sort_values(by='Hash Rate/t')
hash_rate = hash_rate.rename(columns={'Hash Rate/t': 'Timestamp', 'Hash Rate/v': 'Value'})

################ REMOVING BEFORE 2009 E 2022
hash_rate = hash_rate[~(hash_rate['Timestamp'] < '2010-01-01')]
hash_rate = hash_rate[~(hash_rate['Timestamp'] > '2021-12-31')]

#fixing index
hash_rate.index = hash_rate['Timestamp']

And here is the link to download data: Data

EDIT: Most of you are suggesting that the wrong part is that I am doing prediction for a too long time, and that the predictions can be correct for a small period of time. So, here is the prediction using 99% as training set and 1% as test set, so less than 50 days. As you can see is still a straight line. enter image description here Moreover, some of you are saying that (S)ARIMA models are not able to fit data with 2 observations per day, while some others say that (S)ARIMA can. So now I am confused... however I am going to try also after a resampling. However, in my opinion the wrong part is in the prediction.

EDIT: I can confirm that the prediction part is the wrong one. Indeed, I have used the code of another notebook I found online, and here are the results: enter image description here This is the new function used for fitting and prediction. You can compare with the previous one:

# Create list of x train valuess
history = [x for x in x_train]

# establish list for predictions
model_predictions = []

# Count number of test data points
N_test_observations = len(x_test)

# loop through every data point
for time_point in list(x_test.index[-185:]):
    model = sm.tsa.arima.ARIMA(history, order=(1,1,1))
    model_fit = model.fit()
    output = model_fit.forecast()
    yhat = output[0]
    model_predictions.append(yhat)
    true_test_value = x_test[time_point]
    history.append(true_test_value)
        
MAE_error = mean_absolute_error(x_test, model_predictions)
print('Testing Mean Squared Error is {}'.format(MAE_error))
$\endgroup$
9
  • 3
    $\begingroup$ I wouldn't call it completely wrong--Actual vs predicted looks pretty good for the first 100 points (days?). Perhaps that is as well as can be done in this system. $\endgroup$
    – Dave X
    Sep 30, 2022 at 12:45
  • 1
    $\begingroup$ In my opinion it is completely wrong, and there is a problem in the prediction function. Because also using some very simple function in order to test, I obtain only straight lines $\endgroup$
    – CasellaJr
    Oct 2, 2022 at 7:02
  • $\begingroup$ A 70:30 split on looks like 2010-2018.4:2018.4-2022, which looks suspiciously like your test data plot. 100 or 50 days worth of forecast would be more than excellent in a weather, stock, financial, or marketing system. 4 years would be incredible. I think you are extrapolating far beyond what the system appears to support. At what time horizon do the simple functions (e.g. y_hat = b+m*y_{t-100} ) outperform the more complex models? What decision are you trying to make with this data? Bitcoin daytrading? Bitcoin miner hardware prices? Capitol investment? $\endgroup$
    – Dave X
    Oct 2, 2022 at 15:12
  • $\begingroup$ @DaveX I edited the origina post with a prediction of more or less 50 days, and now you can see that it is completely wrong. My idea is to make predictions of an acceptable quantity of days of blockchain data, because I want to use them to analyze the future price of BTC $\endgroup$
    – CasellaJr
    Oct 3, 2022 at 10:15
  • $\begingroup$ @CasellaJr There seems to be high amount of variability in the data, As suggested by others, Please smooth the data. Your data also seems to have structural breaks in it (Please confirm it), Have you taken care of it? If not, Please do take care of it. $\endgroup$ Oct 3, 2022 at 12:18

4 Answers 4

7
$\begingroup$

I looks like youe have two observations per day, is that correct? Before using an ARIMA model with seasonality=7 (weekly seasonality) you need to transform your time series in a way that you only have one observation per day. ARIMA-like models are not suitable for intra-day time series.

Could you share ACF/PACF plots of the transformed time series and/or share your data?

$\endgroup$
6
  • $\begingroup$ Yes, you are right, I have 2 observations per day, how did you noticed? I am going to edit the original post with my data. $\endgroup$
    – CasellaJr
    Sep 29, 2022 at 14:06
  • $\begingroup$ you posted a table with timestamps, each day had 2 entries $\endgroup$
    – Qqbt
    Sep 30, 2022 at 11:11
  • 2
    $\begingroup$ (S)ARIMA won't have any problems with two observations per day. We just have to set the frequency to 14 if we suspect weekly seasonality. I would just not expect bitcoin hash rates to exhibit any seasonality. Also, the data only has one entry per day at the end of the file. It looks like some data cleansing may be indicated. $\endgroup$ Sep 30, 2022 at 17:05
  • $\begingroup$ @StephanKolassa so in case for bitcoin hash rate there is no seasonality, then I will have to use ARIMA, rather than SARIMA, right? $\endgroup$
    – CasellaJr
    Oct 2, 2022 at 7:01
  • 1
    $\begingroup$ Yes, I would expect a nonseasonal approach to work better. However, I don't think ARIMA is a useful model for this kind of data in the first place, because of the inherent multiplicative dynamics. If at all, take logs first. Alternatively, try Exponential Smoothing. I recommend this free online forecasting textbook. $\endgroup$ Oct 2, 2022 at 7:12
4
$\begingroup$

Before testing on 1200+ timesteps, you should test on 100 or 200 timesteps (with sampling or deducing data), because ARIMA have logical constraints.

Then, I suggest to use a simpler model of ARIMA with few parameters before increasing complexity.

If your data is known to have important noise (ex: stock markets), data smoothing could be interesting to improve predictions.

model = ARIMA(df, order=(2,1,2))
results = model.fit(disp=-1)

Note: LSTM is able to memorize more data than ARIMA.

https://towardsdatascience.com/machine-learning-part-19-time-series-and-autoregressive-integrated-moving-average-model-arima-c1005347b0d7

$\endgroup$
4
  • $\begingroup$ Maybe I can use 95% of training set and only 5% as test set, so in this way I test only on less timesteps. I will search for this data smoothing, which I do not know $\endgroup$
    – CasellaJr
    Sep 29, 2022 at 14:05
  • 1
    $\begingroup$ It is not a matter of training data but of data quantity: ARIMA has low performance in long term forecast. capitalone.com/tech/machine-learning/understanding-arima-models $\endgroup$ Sep 29, 2022 at 14:10
  • $\begingroup$ Yes, I mean basically, rather than removing data, I can forecast on few data changing the ratio between training and test $\endgroup$
    – CasellaJr
    Sep 29, 2022 at 14:11
  • 2
    $\begingroup$ Why not, but the test set should also be representative enough. $\endgroup$ Sep 29, 2022 at 14:42
2
$\begingroup$

You did not calculate the uncertainty in your models. Typically in forecasts uncertainty grows exponentially. There is a reason you can't predict daily weather a year in advance.

$\endgroup$
1
$\begingroup$

I agree with @ahavens and others: calculating uncertainty can help to assess, how good or bad your model is. In other words: add also prediction intervals to your graphs. For an ARIMA(p,1,q) model the long term prediction is always a straight line and prediction interval widens as time passes. There is a chance that actual values will be inside that confidence interval. Of course very wide prediction intervals might render a forecast useless, but (just as others have already commented) forecasting this type of data might be indeed very hard. If you also "undo" the log transformation, prediction intervals will grow exponentially. Just for reference: the long term forecast of an ARIMA(p,0,q) model is a straight flat line with a constant prediction interval.

Also I think the forecasts of that "other notebook" you are plotting are just one-step-ahead forecasts, no wonder they fit actuals more precisely than a 1000-step-ahead forecast.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.