1
$\begingroup$

This is related to this article: https://towardsdatascience.com/forecasting-of-periodic-events-with-ml-5081db493c46

I found it interesting and tried to replicate it, having as a result a xgboost classifier that only predicts zeros (not much usefull).

Disclaimer before I post the code: This might be just a dumb issue in which my floats (prediction column) are being truncated in the print, or might be a way-too-byased dataset, or maybe a incorrect algorithm. After the code, I'll post the graphic representation of the decission tree and explain why I cannot make sense out of it.

The code (a bit extense, sorry for that, but for the sake of completiness)

import pandas                   as pd
import numpy                    as np #added by me

import datetime
from dateutil.relativedelta     import relativedelta

import xgboost                  as xgb
from sklearn                    import metrics
from sklearn.model_selection    import train_test_split, GridSearchCV
from sklearn.neighbors          import KNeighborsClassifier
from sklearn.ensemble           import RandomForestClassifier
from xgboost                    import plot_tree
import matplotlib.pyplot        as plt
import os
#this code is necessary, to plot() the tree 
os.environ["PATH"] += os.pathsep + 'C:/Program Files/Graphviz/bin/'

#data with 1's in the prediction (software-release date)
data                    =   pd.DataFrame({'Date': ['2021-01-26','2020-12-22',
                                                    '2020-11-24','2020-10-27','2020-09-29',
                                                    '2020-08-25','2020-07-28','2020-06-30',
                                                    '2020-05-26','2020-04-28','2020-03-31',
                                                    '2020-02-25','2020-01-28','2019-12-31',
                                                    '2019-11-26','2019-10-29','2019-09-24',
                                                    '2019-08-27','2019-07-30','2019-06-25',
                                                    '2019-05-28'
                                                   ]
                                 })
                     
data['Date']            =   pd.to_datetime(data['Date'])
data['Release']         =   1

#expand the gaps
r                       =   pd.date_range(start=data['Date'].min(), end=data['Date'].max())
data                    =   data.set_index('Date').reindex(r).fillna(0.0).rename_axis('Date').reset_index()
#set features
data['Month']           =   data['Date'].dt.month
data['Day']             =   data['Date'].dt.day
data['Workday_N']       =   np.busday_count(data['Date'].values.astype('datetime64[M]') , data['Date'].values.astype('datetime64[D]'))
data['Week_day']        =   data['Date'].dt.weekday
data['Week_of_month']   =   (data['Date'].dt.day - data['Date'].dt.weekday - 2) // 7 + 2
data['Weekday_order']   =   (data['Date'].dt.day + 6) // 7              #counting of weeks, assuming day 1 of the month is always monday. Pretty synthetic feature
data                    =   data.set_index('Date')     


x_train, x_test, y_train, y_test    =   train_test_split(data.drop(['Release'], axis=1), data['Release'] , test_size=0.3, random_state=1, shuffle=False)

xgb_model                           =   xgb.XGBClassifier(  objective           =   'reg:squarederror', 
                                                            colsample_bytree    =   1, 
                                                            learning_rate       =   0.1,
                                                            max_depth           =   4, 
                                                            alpha               =   0.5, 
                                                            n_estimators        =   200
                                                        )
xgb_model.fit(x_train, y_train)
#plot_tree(xgb_model)
#plt.show()
xgb_prediction                      =   xgb_model.predict(x_test)
x_test['Prediction']                =   xgb_prediction
#this displays a bunch of 1's
print("XGBOOST TRAIN  PREDICTIONS:")
with pd.option_context('display.max_rows', None, 'display.max_columns', None,'display.expand_frame_repr', False,'display.float_format','{:,.6f}'.format):  # more options can be specified also
    print(x_test)

print('--------------------------------------------------')
print('--------------------------------------------------')
print('--------------------------------------------------')

#now, let's test it with future data
x_predict                           =   pd.DataFrame(pd.date_range(datetime.date.today(), (datetime.date.today() + relativedelta(years=1)),freq='d'), columns=['Date'])
x_predict['Day']                    =   x_predict['Date'].dt.day
x_predict['Workday_N']              =   np.busday_count(x_predict['Date'].values.astype('datetime64[M]') , x_predict['Date'].values.astype('datetime64[D]'))
x_predict['Week_day']               =   x_predict['Date'].dt.weekday
x_predict['Week_of_month']          =   (x_predict['Date'].dt.day - x_predict['Date'].dt.weekday - 2)//7+2
x_predict['Weekday_order']          =   (x_predict['Date'].dt.day + 6) // 7
x_predict['Month']                  =   x_predict['Date'].dt.month
x_predict                           =   x_predict.set_index('Date')

prediction                          =   xgb_model.predict(x_predict) 
prediction_df                       =   x_predict
prediction_df['Prediction']         =   prediction


#this ONLY displays 0's
print("XGBOOST FUTURE PREDICTIONS:")
with pd.option_context('display.max_rows', None, 'display.max_columns', None,'display.expand_frame_repr', False,'display.float_format','{:,.6f}'.format):  # more options can be specified also
    print(prediction_df)

So, first issue is that it only predict/displays 0's. Not quite sure if predicion is different from 0's, but it is displayed as 0's.

Second issue is that I cannot make sense of the prediction: according to the graphic representation of the decission tree, for each Week_of_month < 4.5 it should return -0.049711, and I obtain 0's.

enter image description here

Thank you SO much, and excuse the extension of the code!

$\endgroup$
3
+50
$\begingroup$

For first issue, Please check the result after making column order of prediction dataset same as training dataset(Date,Month,Day,...) You can check this link

You have specified objective of regression in XGBClassifier. If you correct the order of the column in the prediction data set and change the objective as 'binary:logistic',the graphic representation of tree will be like this Graphical representation of tree Sample prediction : 2021-10-26 10 26 17 1 5 4 1.000000

Week_of_month = 5 , Week_day = 1, Day = 26. For this prediction, leaf value is 0.13333334.And prediction will be one if sigmoid of leaf value is greater than 0.5. Sigmoid of this leaf value is 0.533

$\endgroup$
3
  • $\begingroup$ Good catch on the objective, but apparently it's not used in the sklearn API: github.com/dmlc/xgboost/blob/… $\endgroup$ Oct 21 at 14:22
  • $\begingroup$ +1 for the column ordering. I get an error saying as much when I run OP's code, but maybe older versions of xgb don't? Fixing the column ordering I do indeed get some output class predictions of 1, although as I discuss in my answer one tree might not be enough to confirm that. $\endgroup$ Oct 21 at 14:24
  • $\begingroup$ Silly me... I wouldn't have found I was passing incorrect column order (neither I knew I couldn't do such a thing). Thank you SO much! $\endgroup$
    – glezo
    Oct 24 at 10:01
0
$\begingroup$

The predict method produces the class prediction, which will be either 0 or 1 here. With an imbalanced dataset, it's not unusual for the class predictions to be nearly or entirely the majority class, because the cutoff probability is 50%. You can check predict_proba for the estimated probabilities of each class.

xgboost is an ensemble of trees, so the output of just one of those trees doesn't tell you much. You would need to add the leaf values for every tree to find the aggregate prediction for a given sample, and even then for classification with log-loss objective that aggregate will be a log-odds measurement, not a hard class. The cutoff probability of 50% is a log-odds of 0, so you can tell the class prediction by the sign of the aggregate; so, while you haven't checked the rest of the trees, the negative value in the leaf does support the class prediction of 0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.