# Results are too good.. what is wrong? How to predict correctly?

I am about to evaluate a neural network and want to check whether the predictions make sense.

The variables:

X_train1.shape
> (1700, 2)

X_val1.shape
> (339, 2)

X_test1.shape
> (701, 2)


hold in [:, 0] the x-values (in this case time formats respectively hour-values) and in [:, 1] the corresponding y-values (in this case current values):

X_train1[0:3]
> array([[-1.72694903,  0.63332341],
[-1.71673039,  1.00769389],
[-1.70651176,  1.14177968]])


You can't see time formats in the first column because I had to convert the time formats to numeric values according to

df_train1["timestamp"] = pd.to_timedelta(df_train1['timestamp']).dt.total_seconds()


Beware that I want to make use of an AutoEncoder, where X=Y:

y_train1 = X_train1


and visually:

First, I search the best hyperparameters:

grid_result = grid_obj.fit(X_train1, y_train1)


Then I use the best parameters:

grid_best = grid_result.best_estimator_


Now I want to predict:

prediction = grid_best.predict(X_test1)


in principle this works but I get way too good results though I can see deviations. But I wonder if my concept is meaningful. For example, all the time before, I used predict() to predict new y on X. When I try to do the same above, means:

grid_best.predict(X_test1[:, 0])


Error when checking input: expected input_152 to have shape (2,) but got array with shape (1,)

These are the results:

I train on a period of time of 3 months and predict on 1 month. Why I think that's too good is because I only use five neurons here: (Input 2, latent space 1, output 2). When I use six neurons, it is almost perfect..

• how are you measuring error? The time series looks fearly easy to predict. It shouldn't be surprising that a regressor fits well. Try to compare the error you get with a baseline metric, I'd suggets MASE error adjusting the seasonal parameter to your problem. – lsmor Nov 5 '19 at 11:57
• Where exactly do I define how to measure the error (for prediction)? Either it is negative mean squared error or explained variance, I'm just not sure which exact argument that is. In principle you're right. The current follows a characteristic pattern, I'm just astouned that so few neurons are so powerful and that they are able even to reconstruct few very characteristic structures. Thanks for the advice! – Ben Nov 5 '19 at 13:12
• For anomaly detection I've used statistical model such as Arima or STL decomposition which give a confidence interval. I'd go that way, but you can use also ML methods. There is some literature about this. On the topic of the error, be careful. Time series are tricky, and good predictors might be bad ones when measure with the right error metric. In you case (hourly data) you can use the following predictor I'm gonna predict the next values as the same as yesterday at the same hour. For this Time series looks like a good predictor. Even better than the NN – lsmor Nov 5 '19 at 13:35
• Thanks for the useful input! I thought about combining Arima with DL as I found that it is used sometimes. In my case I have about between 10 - 30 variables which I'll consider. I just focus on one atm and I'll add the other ones piece by piece. Predicting only in one day steps might be difficult due to trends and so on, or not? – Ben Nov 5 '19 at 14:30
• Thank you! Do you mean the framework/library: en.wikipedia.org/wiki/XGBoost ? – Ben Nov 6 '19 at 6:00