I'm working on a binary classification problem with XGBoost and I have a dataset, which has uneven number of observations per user. For some users there are over 100 observations, whereas for some users there are only a few. The "USER_ID" feature is not used as an input for XGBoost.

More specifically, I'm trying to model user physical activity (data collected from wearable trackers) in respect to sleep quality, and some of the variables are demographical features such as as age and sex, alongside steps, heart rate etc. Considering the differing amount of data collected from users, some user behaviours (such night-shift work) are represented more in the data than others due to the number of observations.

How should I take this into account when working with XGBoost?

USER_ID  AGE  SEX  X1  X2    ...  y
1        20   M    65  3000  ...  1
1        ...  ...  ... ...   ...  0
1        ...  ...  ... ...   ...  1
2        30   F    80  2500  ...  0
2        ...  ...  ... ...   ...  1
3        40   M    77  8000  ...  0

The classes are otherwise balanced and I'm able to get good performance for the classifier.

  • $\begingroup$ can you one-hot encode the user id or did you try to simply add the id as a continuous feature? $\endgroup$
    – Peter
    Jul 22 '19 at 18:33
  • $\begingroup$ I think more context would be helpful here. You might want to get rid of user_ids altogether, or you might want to cluster users based on their other variables. $\endgroup$ Jul 22 '19 at 19:04
  • $\begingroup$ Thanks for the comments @BenReiniger! I edited the post to give more information about the problem. $\endgroup$
    – karppmik
    Jul 23 '19 at 11:57

You may want to consider a stratified cross-validation approach where you specifically undersample the data from users that have lots of observations or oversample from uses with fewer observations.

Another approach which may also work is to duplicate the observations of some of the users with less observations, this would even out the weighting of the different kinds of observations.

-- Adding on to answer the additional questions in the comments

There are a number of ways to undersample the majority or oversample the minority. Here's an overview of how to handle the situation (in general).

To undersample the majority, simply remove some % of the training data that is in the majority for each run.

To oversample the minority, duplicate the values that you want to have a higher weight.

Additionally, there is a parameter for XGBoost (scale_pos_weight) which allows you to set the weight of the different samples, see here. Additionally you could leverage SMOTE from the imblearn library.

  • $\begingroup$ Thanks for the suggestion! Do you have any recommendations on how to do oversampling in this case? I'm only familiar with oversampling when it applies to the output class label. $\endgroup$
    – karppmik
    Jul 25 '19 at 9:25
  • $\begingroup$ Rather than answer in the comments, I've updated the answer with a few links and some more details. $\endgroup$
    – MichaelD
    Jul 25 '19 at 21:28
  • $\begingroup$ Actually, the "scale_pos_weight" parameter refers to the class weight (pos/neg) not the instance weight. Apparently, there's also a possibility to control the sample weights with "sample_weight" parameter by using a list of weights for the instances but I've never used that and I'm not exactly sure, how this would affect the model. Could you be more specific, how SMOTE can be used in this case? $\endgroup$
    – karppmik
    Jul 26 '19 at 13:03
  • $\begingroup$ You are correct that the scale_pos_weight refers to the class not the instance level, the same would be true of SMOTE. So if you are specifically looking to weight instances, then the best approach is to use some kind of specific oversample or undersampling based off of other factors in the dataset. $\endgroup$
    – MichaelD
    Jul 26 '19 at 21:08

So, I ended up using SMOTE but instead of binary class label I chose, the "USER_ID" as label for SMOTE. The results seem promising, the descriptive statistics of the data remained almost the same and the performance of the classifier also improved.

Note that I in my classification problem I created the binary target from a continuous variable using certain cutpoints, so the binary target doesn't exist at this point, as I used only numeric variables in this test, but apparently SMOTE also supports categorical variables.

Can anybody spot any fallacies using this method like this? And should this be done before or after train/test split?

import pandas as pd
from imblearn.over_sampling import SMOTE

df = pd.read_csv("data.csv")
df = df.select_dtypes(include=["number"])
df["USER_ID"] = df["USER_ID"].astype(str)

X = df.drop(["USER_ID"], axis=1)
y = df["USER_ID"]

smote = SMOTE(sampling_strategy="auto", k_neighbors=5)

X_sm, y_sm = smote.fit_sample(X, y)

values = pd.DataFrame(data=X_sm, columns=X.columns)
users = pd.DataFrame(data=y_sm, columns=["USER_ID"])

sampled = pd.concat([users, values], axis=1, sort=False)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.