0
$\begingroup$

I am discovering H2O deeplearning and I would like to have your point of view about the performance that's performed my model on classification problem. Do you think my model is overfitting?

dl_fit2 <- h2o.deeplearning(x = predictors, y = response,
                           training_frame = train,
                           validation_frame = valid,
                           epochs = 200,
                           score_validation_samples=10000,
                           score_duty_cycle=0.025,
                           activation = "RectifierWithDropout",
                           hidden = c(80, 10, 80),
                           hidden_dropout_ratios = c(0.2, 0.2, 0.2),
                           loss = "CrossEntropy",
                           rate=0.01,
                           rate_annealing=2e-6,
                           adaptive_rate = FALSE,
                           momentum_start = 0.2,
                           momentum_ramp = 1e7,
                           momentum_stable = 0.4,
                           nesterov_accelerated_gradient = TRUE,
                           l1 = 1e-5,
                           l2 = 1e-5,
                           max_w2=10
)


MSE:  0.009757329
RMSE:  0.09877919
LogLoss:  0.03527449
Mean Per-Class Error:  0.01219048
AUC:  0.9991871
pr_auc:  0.4974259
Gini:  0.9983743

Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
          NO   YES    Error        Rate
NO     10022   140 0.013777  =140/10162
YES      109 10170 0.010604  =109/10279
Totals 10131 10310 0.012181  =249/20441

Maximum Metrics: Maximum metrics at their respective thresholds
                        metric threshold    value idx
1                       max f1  0.367137 0.987906 212
2                       max f2  0.178219 0.990603 270
3                 max f0point5  0.765904 0.990982 122
4                 max accuracy  0.371864 0.987819 210
5                max precision  0.999979 1.000000   0
6                   max recall  0.001865 1.000000 387
7              max specificity  0.999979 1.000000   0
8             max absolute_mcc  0.367137 0.975640 212
9   max min_per_class_accuracy  0.385816 0.987256 204
10 max mean_per_class_accuracy  0.371864 0.987812 210
$\endgroup$
1
$\begingroup$

In general we would need more info on your data and what you are trying to achieve.

However, with 0.999 AUC, it is very likely you are overfitting (or you have a very simple problem).

$\endgroup$
  • $\begingroup$ Hi @lcrmorin, thx for your reply. I thought that 0.999 AUC was good. I was in the false. What is a good value for AUC? I have 7 inputs and i try to train my model with 2 outputs to determine the class Yes or No $\endgroup$ – user979974 Mar 4 '20 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.