1
$\begingroup$

In have two classes in my data, and one of my features with Gini importance 0.023 (8th highest out of 95 features) shows a p-value of 0.44 when doing a t-test on the values from the two classes. I'm wondering if this makes sense. Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

EDIT: If it's helpful, all of the Gini importances range between 0.0011 to 0.0295, and the p-values ranged from 5.13e-109 to 1. These p-values were seen for both t-tests and rank-sum tests.

$\endgroup$
7
  • $\begingroup$ A couple thoughts that don't qualify as a real answer - (1) t-tests have distributional assumptions (like normality) that may not apply to your data. If your data is not normally distributed you can't really interpret the p-value in an intuitive way, and (2) the Gini is calculated for a non-root node in the tree it is contextual based on the rest of the tree, i.e the feature may be discriminatory in the subset of the data going down that branch of the tree but not for the dataset as a whole. $\endgroup$ – tom Feb 22 '18 at 17:03
  • $\begingroup$ Will confirm that each feature is normally distributed. If not, should I use some kind of non-parametric test? I have several thousand data points matched by propensity score, so should a Wilcoxon signed-rank test work? $\endgroup$ – Randy Feb 22 '18 at 21:36
  • $\begingroup$ Will a sign-rank test work for what? What are you trying to accomplish? $\endgroup$ – tom Feb 22 '18 at 22:13
  • $\begingroup$ If I find that the data for one of the features is not normally distributed, I cannot use a t-test. I'm trying to decide which test I would use instead. I want to see if the data in one class statistically differs from the data in the other class for each feature. $\endgroup$ – Randy Feb 22 '18 at 22:24
  • $\begingroup$ @Randy Has my answer resolved your concerns? Let me know if something is still not clear. $\endgroup$ – aivanov Mar 20 '18 at 14:58
1
$\begingroup$

Intuitively, wouldn't features with high Gini importances have very different values between classes? I'm wondering how to reconcile these two facts.

If I understood you correctly, these two facts perfectly reconcile in case your classes are not separable in one-dimensional feature space.

The following exaggerated toy example should illustrate such situation (and power of RF).

#uncomment the following line to remove all variables
# rm(list=ls())

library(ggplot2)
library(caret)
library(MASS)
library(GGally)
library(dplyr)

SEED <- 123456
SIMS <- 1000

set.seed(SEED)

x.signal <- rbind(
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,0.9,0.9,1),2,2)),
    mvrnorm(n = SIMS/2, c(0,0), matrix(c(1,-0.9,-0.9,1),2,2)))

x.noise <- mvrnorm(n = SIMS, c(0,0), matrix(c(1,0.9,0.9,1),2,2))
y <- rep(c(TRUE, FALSE), each=SIMS/2)
df <- data.frame(
    x1.signal=x.signal[,1],
    x2.signal=x.signal[,2], 
    x1.noise=x.noise[,1], 
    x2.noise=x.noise[,2], 
    y=factor(y))

yName <- "y"
xName <- setdiff(colnames(df), yName)

ggpairs(df, aes(colour=y))

enter image description here

ctrl <- trainControl("cv", number = 5, verboseIter = TRUE)
rffit <- train(df[,xName], df[,yName], 
    method = "rf", trControl = ctrl, tuneLength = 5)

varImpPlot(rffit$finalModel)

enter image description here

## t = 0.061767, df = 997.98, p-value = 0.9508
t.test(filter(df, y=="TRUE")$x1.signal, 
    filter(df, y=="FALSE")$x1.signal,
    var.equal = FALSE, paired = FALSE)

## t = 0.22695, df = 996.04, p-value = 0.8205
t.test(filter(df, y=="TRUE")$x2.signal,
    filter(df, y=="FALSE")$x2.signal, 
    var.equal = FALSE, paired = FALSE)


dplyr::group_by(df, y) %>% 
    dplyr::summarise(mx1=mean(x1.signal), 
    mx2=mean(x2.signal))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.