Following is what I learned about the process followed during building and pruning a decision tree, mathematically (from Introduction to Machine Learning by Gareth James et al.):
- Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
- Apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of α.
- Use K-fold cross-validation to choose α. That is, divide the training observations into K folds. For each k = 1, . . .,K: (a) Repeat Steps 1 and 2 on all but the kth fold of the training data. (b) Evaluate the mean squared prediction error on the data in the left-out kth fold, as a function of α. Average the results for each value of α, and pick α to minimize the average error.
- Return the subtree from Step 2 that corresponds to the chosen value
of α.
My question: Going through the above algorithm means we automatically choose the most optimum tree based on minimum average error. Why then, do we have to select maximum depth as the pruning parameter. Shouldn't optimum depth be decided by the algorithm itself and not us?
If anything, shouldn't we be choosing 'K' here, as in how many-fold validation would we like to do?