3
$\begingroup$

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.

What stopping metric can be used to optimize the sensitivity score on validation set?

I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.

My current set-up uses AUC as stopping metric.

df <- as.h2o(df)
split <- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%
train <- h2o.assign(split[[1]], "train.hex") # 60%
valid <- h2o.assign(split[[2]], "valid.hex") # 20%
test <- h2o.assign(split[[3]], "test.hex") # 20%

x <- setdiff(names(df), "disease")
y <- "disease"

gbm <- h2o.gbm(
  x = x, 
  y = y, 
  training_frame = train, 
  validation_frame = valid,
  ntrees = 10000,                                                            
  learn_rate=0.01,     
  # Stopping parameters                                                    
  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC", 
  sample_rate = 0.8,                                                       
  col_sample_rate = 0.8,                                                   
  seed = 1234,                                                             
  nfolds = 50,
  score_tree_interval = 10                                                 
)
h2o.auc(h2o.performance(gbm, valid = TRUE)) 
$\endgroup$
  • $\begingroup$ I'm not familiar with H2O, but can you use the F1 score? $\endgroup$ – Jurgy Nov 30 '17 at 15:07
  • $\begingroup$ @Jurgy The F1 score is based on the recall & specificity. I'm, at this, only interested in optimizing the recall(=sensitivity). $\endgroup$ – wake_wake Nov 30 '17 at 15:45
  • 3
    $\begingroup$ Optimizing for just sensitivity is a bad idea because an always positive approach will lead to maximum sensitivity: S = TP / (TP + FN), with always positive FN = 0 so you get S = TP / TP = 1. $\endgroup$ – Jurgy Nov 30 '17 at 15:51
1
$\begingroup$

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $F_\beta$, a modified version of $F_1$ where recall is considered $\beta$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated both in terms of recall/precision, and in term of type I /type II error. You then need to select a $\beta$ based on how much recall is more important to you... this can be done by considerations on the costs of type I vs the cost of type II errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.