XGBoost, binary classification: uneven number of observations per user

Question

I'm working on a binary classification problem with XGBoost and I have a dataset, which has uneven number of observations per user. For some users there are over 100 observations, whereas for some users there are only a few. The "USER_ID" feature is not used as an input for XGBoost.

More specifically, I'm trying to model user physical activity (data collected from wearable trackers) in respect to sleep quality, and some of the variables are demographical features such as as age and sex, alongside steps, heart rate etc. Considering the differing amount of data collected from users, some user behaviours (such night-shift work) are represented more in the data than others due to the number of observations.

How should I take this into account when working with XGBoost?

USER_ID  AGE  SEX  X1  X2    ...  y
1        20   M    65  3000  ...  1
1        ...  ...  ... ...   ...  0
1        ...  ...  ... ...   ...  1
2        30   F    80  2500  ...  0
2        ...  ...  ... ...   ...  1
3        40   M    77  8000  ...  0

The classes are otherwise balanced and I'm able to get good performance for the classifier.

Answer 1

You may want to consider a stratified cross-validation approach where you specifically undersample the data from users that have lots of observations or oversample from uses with fewer observations.

Another approach which may also work is to duplicate the observations of some of the users with less observations, this would even out the weighting of the different kinds of observations.

-- Adding on to answer the additional questions in the comments

There are a number of ways to undersample the majority or oversample the minority. Here's an overview of how to handle the situation (in general).

To undersample the majority, simply remove some % of the training data that is in the majority for each run.

To oversample the minority, duplicate the values that you want to have a higher weight.

Additionally, there is a parameter for XGBoost (scale_pos_weight) which allows you to set the weight of the different samples, see here. Additionally you could leverage SMOTE from the imblearn library.

Answer 2

So, I ended up using SMOTE but instead of binary class label I chose, the "USER_ID" as label for SMOTE. The results seem promising, the descriptive statistics of the data remained almost the same and the performance of the classifier also improved.

Note that I in my classification problem I created the binary target from a continuous variable using certain cutpoints, so the binary target doesn't exist at this point, as I used only numeric variables in this test, but apparently SMOTE also supports categorical variables.

Can anybody spot any fallacies using this method like this? And should this be done before or after train/test split?

import pandas as pd
from imblearn.over_sampling import SMOTE

df = pd.read_csv("data.csv")
df = df.select_dtypes(include=["number"])
df["USER_ID"] = df["USER_ID"].astype(str)

X = df.drop(["USER_ID"], axis=1)
y = df["USER_ID"]

smote = SMOTE(sampling_strategy="auto", k_neighbors=5)

X_sm, y_sm = smote.fit_sample(X, y)

values = pd.DataFrame(data=X_sm, columns=X.columns)
users = pd.DataFrame(data=y_sm, columns=["USER_ID"])

sampled = pd.concat([users, values], axis=1, sort=False)