I have a years worth of electricity power data on 15 minute intervals joined with weather data and time-of-week one hot dummy variables.

Is using train/test split an okay approach for validating the model? Am attempting to predict electricity with explainer variables like weather and time-of-week dummies.

For starters, I weeded out a bunch of dummy variables variables with OLS regression in statsmodels and then attempted to fit the model with XG Boost. Would anyone have some tips for a better approach on fitting time series data, validate the ML model, and then attempting to use regression to predict electricity? Some of my Python code for the ML training process:

# shuffle the DataFrame rows
df2 = df2.sample(frac=1)

train, test = train_test_split(df2, test_size=0.2)

regressor = XGBRegressor()

X_train = np.array(train.drop(['total_main_kw'],1))
y_train = np.array(train['total_main_kw'])

X_test = np.array(test.drop(['total_main_kw'],1))
y_test = np.array(test['total_main_kw'])

regressor.fit(X_train, y_train)

predicted_kw_xgboost = regressor.predict(X_test)

y_test_df = pd.DataFrame({'test_power':y_test})
y_test_df['predicted_kw_xgboost'] = predicted_kw_xgboost


Will plot trained model predicting the test dataset but I have not done any verification if the data is stationary or not:

enter image description here

mse = mean_squared_error(y_test_df['test_power'], y_test_df['predicted_kw_xgboost'])
print("MEAN SQUARED ERROR: ",mse)
print("ROOT MEAN SQUARED ERROR: ",round(np.sqrt(mse),3)," IN kW")

MEAN SQUARED ERROR:  4.188126272978789

Thanks any tips still learning in this area..


1 Answer 1

  1. Ideally, your approach for validating the model should be a train-validation-test approach. i.e. the model is trained on the training data, the results of the model are then validated on a set of validation data that is segregated from the training data (e.g. a 70/30 split), and then the model is tested on previously unseen data. This last point is important – the test data must be completely separate from that used to train and validate the model. Otherwise, data leakage occurs and overfitting results – whereby the model gives a falsely high gauge of accuracy and will go on to perform poorly on real-world data.

  2. The time series gives the preliminary appearance of stationarity, but a more formal test such as KPSS could be used to verify this.

  3. The root mean squared error of 2.046 kw does not mean anything by itself. Rather, one should obtain the mean value in the test set, and then compare this to the root mean squared error. For instance, if the mean of the test set is 200 kw, then the RMSE is quite low in comparison indicating strong predictive performance. However, if the mean of the test set were to be 1 kw, then the RMSE is conversely quite high in comparison – indicating weak predictive performance.

  • $\begingroup$ Would you know of any good train-validation-test approach examples to look at with Python using XGboost or scikit Learn? Also curious to ask if using regression shuffling the data is a bad thing to do where it is time series data where I am attempting to use a regression approach $\endgroup$
    – bbartling
    Apr 7, 2023 at 14:57
  • $\begingroup$ You may find this resource useful: machinelearningmastery.com/xgboost-for-time-series-forecasting. With regard to your question, I think you are referring to k-fold cross validation? Yes, "regression shuffling" as you put it is not the right approach, as you are working with sequential data. Instead, walk-forward validation should be used, which is also described in the attached link. $\endgroup$ Apr 7, 2023 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.