I am currently working on my first machine learning model (Palmer Penguins dataset). I am going to train the 3 machine learning models, each of them using the different model architecture (Decision Tree, Random Forest & Gradient Boosting), and compare them with each other. I understand that in my particular case the test / train split will be necessary if I want to compare the accuracy of three different models. But is it always a case that we need to split the dataset into training set and the test set?
Let's take an example of the Random Forest algorithm - we can evaluate our model using the OOB score and perform the actual testing without performing the train / test split. Since we will already have a bunch of samples in our training set that won't be actually used for training I think it could be a good idea to use them for testing, instead of reducing the training set by explicitly splitting it into train / test set. I think such an approach could be specifically useful when we have small datasets (such as Palmer Penguins for example) when every sample that we discard from training could have a big impact on the model performance.
So here is the question again - is test / train split really always necessary? And why is it considered a thing that should be always performed?