Understanding Sklearns learning_curve

Question

I have been using sklearns learning_curve , and there are a few questions I have that are not answered by the documentation(see also here and here), as well as questions that are raised by the function about sklearn more generally

Here are some learning curves from my models of a data set

And the code that produced them:

train_sizes, train_scores, valid_scores =learning_curve(linear_regression_model,rescaled_X_train,Y_train)
axes[0,0].plot(train_sizes,train_scores)
axes[0,1].plot(train_sizes,valid_scores)

train_sizes, train_scores, valid_scores =learning_curve(random_forest_model, rescaled_X_train,Y_train)
axes[1,0].plot(train_sizes,train_scores)
axes[1,1].plot(train_sizes,valid_scores)

1. The documentation makes it seem like, the line learning_curve(linear_regression_model, rescaled_X_train, Y_train) fits the model rather than simply showing how the models fitting process previously behaved?

a. If it is fitting the model again – how do you pass hyperparameters (for example gamma for a SVM or maximum tree depth) and determine the cost function that is being used?

b. If not, this seems very strange. I would have assumed that a linear regressor was by default just fit by least squares rather than something involving k-fold validation, as it appears to be if I am viewing the above graphs correctly. Is this how sklearn normally fits regressors?

2. is the y- axis on these graphs accuracy score?

Answer 1

The term learning curve can mean different things in different context, which is confusing.

When talking about neural networks (and other iteratively trained models) the learning curve describes the model's training progress. It is often used to determine when it's time to stop training.

In scikit-learn, the learning curve is interpreted differently. It describes how your model would perform if it was (re-)trained with less data. This can help you guess if the model would likely improve by getting more data.

The same hyperparameters specified when constructing the model are used when the model is re-fitted.

The score function used is also a parameter of the model. Many regression models default to the R2 score, which is likely the score you plotted.

Answer 2

It is correct that calling learning_curve will refit your model multiple times for different training dataset sizes. You can simply pass specific hyperparameters when initializing the model you want to use, which you can then pass to learning_curve for the estimator argument. The actual loss funtion that is used depends on the type of estimator you are providing, searching the documentation for the specific estimator will probably give you more info on the loss function used when fitting the model.