0
$\begingroup$

So lets assume I have a binary classification problem and I started out with a logistic regression model. I quickly evaluate the models accuracy (lets assume we don't have a skewed dataset). After this I start to implement other models to compare, I begin with checking the accuracy score and plot a ROC curve to evaluate AOC score.

Lets say that I carry on with my logistic regression model. I wan't to fine-tune it and use a grid search algorithm and from there carry on choosing the best hyperparamters w.r.t the accuracy score.

When do I actually plot the learning and validation curve? I pressume that I should plot it next to see how a larger set of paramters affect the score and then to see if my model is underfitting or overfitting.

But shouldn't I plot the learning curve much earlier for example in the first step where I choose to start with a logistic reg. model?

To sum it up: In what stage is it appropiate to plot a learning curve?

Improve this question
$\endgroup$
4
  • $\begingroup$ cross-validated -> plot -> tune -> repeat $\endgroup$
    – Madhur Yadav
    Commented Sep 14, 2020 at 16:07
  • $\begingroup$ Plot what? The ROC or the learning curve and validation curve? $\endgroup$
    – John
    Commented Sep 14, 2020 at 19:27
  • $\begingroup$ metrics v/s iteration for both valid and training. google.com/… $\endgroup$
    – Madhur Yadav
    Commented Sep 14, 2020 at 19:34
  • $\begingroup$ Huh? I now the difference between validation and learning curve $\endgroup$
    – John
    Commented Sep 14, 2020 at 19:44

1 Answer 1

Reset to default
1
$\begingroup$

I understand that with "learning curve" you are referring to the plot of the loss function over the training data (or subsets of it) when optimizing with iterative methods, like gradient descent. This is sometimes referred to as the training loss plot.

You use a training loss plot during training, to evaluate the convergence of your training algorithm and model. Usually, together with the training loss plot, you visualize the validation loss, that is, the loss function computed on a held-out dataset, to evaluate whether your model is overfitting your training data.

Normally, the training loss gets a new point after each optimization step. For instance, in stochastic gradient descent, the training loss points are the loss evaluated at each minibatch. On the other hand, the validation loss curve gets a new point after each epoch, or at least after a few optimization steps have been performed.

This is different from a ROC curve, which is evaluated once the model is fully trained, to understand the behavior of the trained model.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, no I mean "learning curve" it terms of the model’s performance on the training set and the validation set as a function of the training set size not with each step. But to be more clear I am confused about the workflow of a ML binary classifcation problem. Do I evaluate the learning curve before, after or before and after I fine-tune a specific model? And when comparing different classification models should I fine-tune all of them before comparing them to eachother? $\endgroup$
    – John
    Commented Sep 15, 2020 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.