3
$\begingroup$

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.

What stopping metric can be used to optimize the sensitivity score on validation set?

I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.

My current set-up uses AUC as stopping metric.

df <- as.h2o(df)
split <- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%
train <- h2o.assign(split[[1]], "train.hex") # 60%
valid <- h2o.assign(split[[2]], "valid.hex") # 20%
test <- h2o.assign(split[[3]], "test.hex") # 20%

x <- setdiff(names(df), "disease")
y <- "disease"

gbm <- h2o.gbm(
  x = x, 
  y = y, 
  training_frame = train, 
  validation_frame = valid,
  ntrees = 10000,                                                            
  learn_rate=0.01,     
  # Stopping parameters                                                    
  stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC", 
  sample_rate = 0.8,                                                       
  col_sample_rate = 0.8,                                                   
  seed = 1234,                                                             
  nfolds = 50,
  score_tree_interval = 10                                                 
)
h2o.auc(h2o.performance(gbm, valid = TRUE)) 
$\endgroup$
3
  • $\begingroup$ I'm not familiar with H2O, but can you use the F1 score? $\endgroup$
    – Jurgy
    Commented Nov 30, 2017 at 15:07
  • $\begingroup$ @Jurgy The F1 score is based on the recall & specificity. I'm, at this, only interested in optimizing the recall(=sensitivity). $\endgroup$
    – wake_wake
    Commented Nov 30, 2017 at 15:45
  • 3
    $\begingroup$ Optimizing for just sensitivity is a bad idea because an always positive approach will lead to maximum sensitivity: S = TP / (TP + FN), with always positive FN = 0 so you get S = TP / TP = 1. $\endgroup$
    – Jurgy
    Commented Nov 30, 2017 at 15:51

1 Answer 1

1
$\begingroup$

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $F_\beta$, a modified version of $F_1$ where recall is considered $\beta$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated both in terms of recall/precision, and in term of type I /type II error. You then need to select a $\beta$ based on how much recall is more important to you... this can be done by considerations on the costs of type I vs the cost of type II errors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.