Test / Train split - is it always necessary (supervised learning)?

Question

I am currently working on my first machine learning model (Palmer Penguins dataset). I am going to train the 3 machine learning models, each of them using the different model architecture (Decision Tree, Random Forest & Gradient Boosting), and compare them with each other. I understand that in my particular case the test / train split will be necessary if I want to compare the accuracy of three different models. But is it always a case that we need to split the dataset into training set and the test set?

Let's take an example of the Random Forest algorithm - we can evaluate our model using the OOB score and perform the actual testing without performing the train / test split. Since we will already have a bunch of samples in our training set that won't be actually used for training I think it could be a good idea to use them for testing, instead of reducing the training set by explicitly splitting it into train / test set. I think such an approach could be specifically useful when we have small datasets (such as Palmer Penguins for example) when every sample that we discard from training could have a big impact on the model performance.

So here is the question again - is test / train split really always necessary? And why is it considered a thing that should be always performed?

Answer 1

Yes OOB score can be helpful when data is hard to come by. If your dataset size is small you can use OOB to evaluate the model.

But another, much better option would be to split your data and apply nested cv to evaluate the model and do Hyperparameter tuning at the same time. This way even if your dataset size is small, you will utilize the whole data.

Till now I haven't came across an example where the model is built and evaluated using only OOB score.

Someone kindly correct me if I am wrong.

Cheers!

Answer 2

Yes!!!, its important to have separate training and testing data because there is a chance of overfitting if we don't do that (if we don't have separate training and test data).

In general in any machine learning task we deal with data that is a subset from a population ( bigger data set for which we don't have access to!!!).So, if we don't have test data then its highly impossible to tell if how the model works on unseen data, therefore we cant deploy it for general use ( at the end we need to deploy the model for practical use).

I have been machine learning models and have seen that some times models work well on training data -95% to 100% accuracy but when the same model is tested on the test data it showed an accuracy of 45%-60% which is very much low!!!.

Hope I have cleared your doubt... feel free to ping me for further query's...

Answer 3

Yes.

1. It is quite nice in that it saves time right when you need it most (i.e., training takes longer because of more rows).

2. It is especially fine if you are using algorithms that avoid overfitting (like random forest, lasso, etc).

3. You should always validate model accuracy in production anyhow (whether using train/test or CV).