Predict the probability that a user is active on a website

Question

I have been given two different arrays, which describe if a user is active on a site or not based on some features. The first array is for training. Each row of the array is a different user and the columns are different features. The first 896 columns describes the users using-patterns and the remaining 28 are lables (These 28 columns stands for 28 different time periods, they are labled 1 if the user is active in the time period, 0 otherwise.

The second array is for testing and has the same amount of columns, but here the last 28 rows are empty. My task is to predict the probability that a user will be active during each time period. I am a bit lost and are looking for (preferably quite simple) methods to solve this problem with.

Answer 1

I’ll throw out a couple of hints / clarifications.

1. Your ‘test’ set is of no use to you since you have the dependent variables missing, so just ignore it to begin with.
2. You will want to split you ‘training’ data into two sets to begin with, one for training a model and one for testing the model and computing the effectiveness of the model.
3. Hopefully you have lots of records in the training set because you have so many independent variables. Some models are very sensitive to low sample size and high dimensionality.
4. You will need to decide if the feature columns need to be encoded. You’ll have to pay particular attention to any categorical features because they need to be one hot encoded.
5. You will need to choose a model or ensemble of models. You may start by trying a binary classifier. You will need to creat one for each of your timeframes. Each model should take the ~890 features and produce one result with a 0 or 1.
6. You will need to figure out how to evaluate your models to understand how the choices you are making to tune each model are influencing the results.

This is not a comprehensive list, but hopefully it is helpful to get you started.

Answer 2

You are looking for a model like: $y = \beta X + u$, where $y$ is one or more of the 28 different time periods and $X$ is a feature matrix containing all or some of the 896 columns which describe the users using-patterns. Since $y$ is binary, your problem is classification.

Since you say you look for a simple approach, I could imagine a logistic regression. Here is a minimal example in R:

# Data
df = data.frame(c(12,32,23,13,45,31), c(657,456,265,263,475,354), c(8,5,9,4,6,3), c(1,1,0,0,1,0), c(0,1,1,0,0,1))
colnames(df) = c("f1", "f2", "f3", "t1", "t2")
df

# Logit
mylogit1 <- glm(t2 ~ t1+f1+f2+f3, data = df, family = "binomial")
summary(mylogit1)

# Predict outcome
preds = predict(mylogit1, newdata = df, type = "response")

# Look at AUC
library(Metrics)
auc(df$t2, preds)

# Look at the confusion matrix
library(caret)
preddf = data.frame(as.factor(round(preds)),as.factor(df$t2))
colnames(preddf)=c("pred", "truth")
confusionMatrix(preddf$pred, preddf$truth)

First I generate some fake data with three features f and two targets t. It looks like:

  f1  f2 f3 t1 t2
1 12 657  8  1  0
2 32 456  5  1  1
3 23 265  9  0  1
4 13 263  4  0  0
5 45 475  6  1  0
6 31 354  3  0  1

Second I run a logistic regression where I use t2 as $y$ (this is what I predict) and f1,f2,f3 as features $X$ (what I use to make a prediction):

    Call:
glm(formula = t2 ~ f1 + f2 + f3, family = "binomial", data = df)

Deviance Residuals: 
      1        2        3        4        5        6  
-0.2496   1.3025   0.7380  -1.2315  -1.4095   0.9806  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)
(Intercept)  1.443719   4.298364   0.336    0.737
f1           0.071943   0.111565   0.645    0.519
f2          -0.009584   0.012244  -0.783    0.434
f3           0.066979   0.471445   0.142    0.887

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 8.3178  on 5  degrees of freedom
Residual deviance: 6.7684  on 2  degrees of freedom
AIC: 14.768

Number of Fisher Scoring iterations: 5

No surprise, the regression does not make too much sense given the fake data.

However, now we can make predictions and test if our model works well. Usually you would use some part of the data which has not been used for model training to check how well your model performs. Here I use just the same data for convenience.

You can look at the confusion matrix for instance to check how many correct predictions you have made:

Confusion Matrix and Statistics

          Reference
Prediction 0 1
         0 1 1
         1 2 2

Well, not too good in this case, just 50% correct.

So you could use such a model to predict each of the targets (time periods) you have in your data.

There are of course a lot more options how to model this, e.g. "boosting" or you could also predict all target at once in a multi-target model. You can also look for an improved representation of your features $X$, e.g. in a generalised additive model. Or you can use a "lasso" to shrink features in $X$ which are not useful for prediction.

Just make sure you use a logistic link function to predict binary outputs.