I'm using Sklearn to build a classifier in which my client wants a predicted probability for each row of data. The default of let's say Random Forest is if > 50%, then classifies as TRUE, but using the predict_proba function I'm able to get the probability.
The data I'm given has 10k rows, which all are labeled TRUE or FALSE. If it's my job to provide a predicted probability by row, should I still use a 70/30 Train/Test split. In which I create my best model on the Train 70% using a 10-fold CV. But then when I need to output for all 10k rows, I would actually be make a prediction on the training data and the test data, which doesn't seem correct to predict on training data. Any thoughts on how to approach this?