2
$\begingroup$

i tried to evaluate 6 models and after plotting , this what i get : enter image description here

So i'm wondering , if those results are "Right" ?

Thank's in advance.

$\endgroup$
  • $\begingroup$ Can you give some more information on your project? As Juan said, these AUC numbers are suspiciously good, and this chart alone is not very informative. $\endgroup$ – Upper_Case May 21 at 15:36
  • $\begingroup$ I have to develop a model to predict whether a person will get hired as DS or not , and here is just some model evaluation , to see what's the best model to use for my task . $\endgroup$ – Dimi May 21 at 15:43
5
$\begingroup$

Did you evaluate the results in the training set? Or in the test set?

Those results are outstandingly good! Suspiciously good.

I think you tried your results in the training set only, so your results reflect overfitting on your data, which means your model learned the set, it was not generalized (which means it is unapplicable to any other dataset you may encounter in the future, which is not useful).

For comparing ROC between methodologies you should model them being careful for overfitting and try them on a test dataset (a dataset which you never knew before, which can be obtained partitioning your dataset).

In that way your comparison is not measuring which model learns by memory your data.

$\endgroup$
  • $\begingroup$ Well , I did split my data into Training and testing and this evaluation has been done only on the Training. $\endgroup$ – Dimi May 21 at 15:37
  • $\begingroup$ Yes! That is what happens! You should assess your model in the dataset your model doesn't know (test dataset), so you can measure the generallity of it. $\endgroup$ – Juan Esteban de la Calle May 21 at 16:01
  • $\begingroup$ Alright , thank's a lot $\endgroup$ – Dimi May 21 at 16:06
2
$\begingroup$

I'm not sure that AUC is the right value to use to compare these models. Have a look at this question for a bit more detail.

In any case, the AUC of your training data is not a very informative piece of information, and assessing the performance of your model on the training set isn't enough to determine how "right" your model(s) may be, no matter how you go about it. Such a comparison could be done on a test set, at the earliest, and better still, totally out-of-sample data (as final model training usually includes re-training on the full data set).

Finally, model performance (in application) will be defined by choosing a specific cut point for your predictors (and the corresponding true positive/false positive tradeoff) rather than by looking at the overall ability of your model to discern between outcomes across many possible cut points. Overall AUC may still be interesting, but applied model performance is a more precise question (this question has some good information on that).

$\endgroup$
  • $\begingroup$ Intresting , Thank you $\endgroup$ – Dimi May 21 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.