1
$\begingroup$

I have a dataset with three columns, the number of days behind or ahead of the deadline a project was completed, the name of the department whose project it was and a user number for the employee.

The company are interested in predicting when work is likely to be completed, given the department and the employee.

Looking at these box graphs you can see that there is a slight difference in project delivery delay between departments and for individuals but that there is a vastly bigger difference in variance between departments and individuals.

Largely because of this, if I use statistical tests I find that there is no significant difference between the means of the classes.

However I can build a complex decision tree or similar which attempts to predict the delay on a project based on the user/ department.

  • Is this meaningful?
  • Under what circumstances would it be meaningful to use ML tools on a data set that statistical tools would reject the idea of a pattern?
  • If you use ML tools without statistics, when would you reject the idea that there is any kind of significant pattern in your data?

Delay for projects from each department

Delay for projects from each department^

Delay for projects for each employee

Delay for projects for each employee^

Improve this question
$\endgroup$
2
$\begingroup$

It depends on what kind of difference you examine statistically.

If you have two groups drawn from the same distribution, you should not be able to distinguish between them reliably. This is a feature, not a bug, of machine learning.

If you have two groups drawn from different distributions that have the same mean, you should be able to distinguish between them. Your performance might not be great, but you can do better than randomly guessing. For example, if you include a quadratic term in a logistic regression, you can distinguish between $N(0,1)$ and $N(0, 2)$. Your performance will not be amazing, but it beats randomly guessing.

# install.packages('pROC') # You might need to run this.
library(pROC)
set.seed(2021)
N <- 100
x0 <- rnorm(N, 0, 1)
y0 <- rep(0, N)
x1 <- rnorm(N, 0, 2)
y1 <- rep(1, N)
x <- c(x0, x1)
y <- c(y0, y1)
L1 <- glm(y ~ x, family = binomial)
L2 <- glm(y ~ poly(x, 2), family = binomial) # I think poly is not the best way to do this, but it's what I remember right now.
my_roc_1 <- pROC::roc(y, 1/(1+exp(-predict(L1))))
my_roc_2 <- pROC::roc(y, 1/(1+exp(-predict(L2))))
par(mfrow = c(2, 1))
plot(my_roc_1, main = "No Quadratic Term")
plot(my_roc_2, main = "Quadratic Term")
par(mfrow = c(1, 1))

enter image description here

Without the quadratic term, the performance is terrible, and the AUC is basically chance, at $0.5426$. With the quadratic term, the performance is far from amazing, but we do better than chance, with an AUC of $0.6847$. (Even better could be to compare the log loss or Brier score, though those metrics seem less popular in data science than in statistics.)

If you plot the densities of the two groups, it is clear why they can be distinguished better than chance.

library(ggplot2)
d0 <- data.frame(x = x0, group = "0")
d1 <- data.frame(x = x1, group = "1")
d <- rbind(d0, d1)
ggplot(d, aes(x = x, fill = group)) + geom_density(alpha = 0.3) + theme_bw()

enter image description here

If you get a point near $0$, it is more likely to be from group $0$ than group $1$. If you get a point far from $0$, it is more likely to be from group $1$ than group $0$. Around $\pm2$, the group is more ambiguous, but you can do decent when you are in the middle or way out far.

The lesson to learn here is that some model should be able to make some progress in distinguishing between the two distributions, unless the distributions are identical. Linear methods like vanilla logistic regression (with just linear terms, so no $x^2$ like my L2 model) work best when the means are different. A logistic regression with nonlinear terms, such as my L2 model that has a quadratic term or a model with splines, can fit (but also overfit) other differences. In the extreme, a model that finds its own nonlinear terms, such as a neural network, can find all kinds of differences (while also risking overfitting).

Getting back to your example, you know that if you make an extreme observation, it is more likely to belong to some groups that have extreme observations than other groups that are clustered near the mean.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I cannot find the article now, but I remember reading a paper about using machine learning methods to do distribution tests (even in the multivariate setting). The upside would be the ability for something like a neural network to find bizarre, nonlinear differences between the groups, differences that might not be apparent from more routine hypothesis tests. In that sense, the machine learning model conducts the hypothesis test. (This is not so different in spirit from hypothesis testing nested generalized linear models, such as an ANOVA F-test.) $\endgroup$
    – Dave
    Aug 5 at 17:35
  • $\begingroup$ Thanks for the in depth response - do you think given the above data it would be meaningful to predict how early or late a project will be done given a user id and a department given the data? $\endgroup$
    – Abijah
    Aug 6 at 14:03
  • 1
    $\begingroup$ If user ID is repeated, you might make some progress by noticing that certain users are consistently early or late. Otherwise, the means all look to range from a little early to a little late, which might not be the most valuable business insight. // Depending on how sophisticated you want to get, you might be able to get more insight from confidence intervals, prediction intervals, or conditional quantiles (quantile regression). $\endgroup$
    – Dave
    Aug 6 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.