# Propensity Modeling, still use Test/Train Split?

I'm using Sklearn to build a classifier in which my client wants a predicted probability for each row of data. The default of let's say Random Forest is if > 50%, then classifies as TRUE, but using the predict_proba function I'm able to get the probability.

The data I'm given has 10k rows, which all are labeled TRUE or FALSE. If it's my job to provide a predicted probability by row, should I still use a 70/30 Train/Test split. In which I create my best model on the Train 70% using a 10-fold CV. But then when I need to output for all 10k rows, I would actually be make a prediction on the training data and the test data, which doesn't seem correct to predict on training data. Any thoughts on how to approach this?

In the outer cross-validation steps, you break your data in $N_{out}$ folds, where one fold is kept for testing the model and the remaining $N_{out} - 1$ folds are used to train. You then feed this training dataset (which is similar, in nature, to that 70% you have mentioned in OP) to the inner cross-validation step with $N_{in}$ folds. Here, you will compare different models and choose the best performing one. This chosen model is then passed to the outer fold (see above) and can be used to obtain probabilities for the 1 fold that was left out. You then repeat the same procedure for other folds in the outer fold. At the end of this nested cross-validation process, you will have probabilities for all your rows (but they will have come from $N_{out}$ different classifiers, each corresponding to one of the outer folds).