# Classifier Performance - Binomial proportion confidence interval

I am solving a binary classification task.

As an output of Random Forest classifier, I get a probability of how sure RF is that class is 0 or a 1.

How can I calculate the needed threshold, to be 95% sure that calculated output is really a class my RF predicted.

(I do not want to split the data at 0.5, rather I would take class 0 if it is lower than 0.3 and class 1 if it is higher than let say 0.7. Although I would like to get the least spread and get 95% of right data choices for 0 and 1).