I have a dataset with 4519 samples labeled as "1", and 18921 samples labeled as "0" in a binary classification exercise. I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class.
However, should the test dataset be balanced as well?
In other words, if train my model with 1000 random samples of "0" class, and 1000 random samples of "1" class, should I test the model with the remaining 3519 samples of "1" class, and randomly select another 3519 samples of the majority "0" class, or I can go with the remaining 17921?
What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?