0
$\begingroup$

I have performed logistic regression. And I am getting an accuracy of 77% with my current model. I divided my training set into cross validation set and train set. And I plotted a learning curve (graph of training examples vs cost function for train set and cost function of cross validation set). My learning curve is shown below - enter image description here

What does it indicate? Since both the curves have a very less difference one is (0.51 and other in 0.52), Is my model biased, or is it correct?

$\endgroup$
4
  • 1
    $\begingroup$ It is weird that your train cost increases with the number of samples. $\endgroup$ May 29 at 13:26
  • 1
    $\begingroup$ Welcome to DataScienceSE. It looks like there's an error in the Y label of the graph, this doesn't look like a cost, it's more likely a measure of performance but we don't know which one? Apparently it's not accuracy since you said that the accuracy is 77%. Also it's not clear what is your "cross-validation set", it could be a validation or test set but then I don't understand how/why its performance changes with the number of instances. $\endgroup$
    – Erwan
    May 29 at 22:15
  • $\begingroup$ I just realized that you might have a confusion: are you aware that this graph shows the result of an ablation study, according to its X label? this means training a model many times with different number of instances. If you're just trying to compare the performance between training and test set in order to check for overfitting, you don't need this kind of graph. However you need to know which evaluation measure you're using ;) $\endgroup$
    – Erwan
    May 29 at 22:18
  • 1
    $\begingroup$ In case you actually intended to do an ablation study, I think the problem is that the test set (what you call the cv set) performance shouldn't be done like this: apparently you just evaluate the final model trained with the full data, but only on a number of test instances X. It should be the opposite: the model is applied every time on the full test set, but the model is the one obtained with X instances in the training set. $\endgroup$
    – Erwan
    May 29 at 22:23
1
$\begingroup$

Generally speaking, the "normal" shape of a learning curve (defined as a "plot of error vs training set size is known as a learning curve" (1)) is to observe an initially very low training error indicating that the model almost perfectly learns the small amount of training data while the test error will be high. When the amount of training data increases, the training error is expected to increase, too, as it becomes harder for the model to learn the increasingly complex data. At some point usually the training error stops increasing because data complexity, i.e. the number of distinct patterns in the data, does not increase further - even when adding more data.

In contrast, the test error is expected to be high in the beginning and then decrease when train and test data become more similar (since you're adding more training data). That is, in the beginning the model overfits (which is good news since it means that it's able to learn the data) and later train and test error ideally converge. This occurs when training and test become more similar.

This could look similar to this (The black horizontal line is the Bayes error) (1):

MSE on trai/test sets vs size of training set, data generated from a degree 2 polynomial with Gaussian noise w/ Var = 4. Polynomial models of varying degree fit to this data.

Or like this for a more complex model (1):

MSE on trai/test sets vs size of training set, data generated from a degree 2 polynomial with Gaussian noise w/ Var = 4. Polynomial models of varying degree fit to this data.

And that is similary to what your graph shows.

In contrast, a model which is not able to capture even simple patterns when the amount of training data is low could produce a learning curve like this (1):

MSE on trai/test sets vs size of training set, data generated from a degree 2 polynomial with Gaussian noise w/ Var = 4. Polynomial models of varying degree fit to this data.

A third scenario would be a very complex model showing a learning curve like this (1):

MSE on trai/test sets vs size of training set, data generated from a degree 2 polynomial with Gaussian noise w/ Var = 4. Polynomial models of varying degree fit to this data.

Compared to the first plot, the training error does not increase as fast since the more complex model is able to overfit data even when data complexity increases. But it also shows a very high test error in the beginning and a larger gap between train and test error for larger amounts of data.

This answer might also be interesting for you to read.


References: (1) Probabilistic Machine Learning: An Introduction p. 109-110

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.