overfitting of regression tree from the beginning

Question

I am trying to build a regression tree to model insurance claim frequencies. I have 36000 observations and 9 covariates.

My model overfits right from the beginning!, ie as the cost complexity goes down, the relative validation error goes up. I am using this command in R:

rpart_model <-rpart(cbind(Duration,Nbclaims) ~ Gender + DriverAge + 
  CarAge + Area + Leasing + Power + Fract + Contract, data = motor,    method="poisson",parms=list(shrink=1), control=rpart.control(cp=0))

and obtain these results:

        CP nsplit rel error xerror     xstd
1   1.9019e-03        0     1.00000   1.0002   0.022911

2   1.3854e-03        1     0.99810   1.0004   0.022973

3   1.1587e-03        2     0.99671   1.0057   0.023185

4   9.1009e-04        5     0.99324   1.0069   0.023269

5   9.0852e-04        6     0.99233   1.0134   0.023526

6   8.8781e-04        7     0.99142   1.0134   0.023531

7   8.5397e-04       10     0.98872   1.0150   0.023603

8   8.5119e-04       11     0.98786   1.0153   0.023621

I have done my research to try and figure this issue out but am completely stuck for the moment, could anyone help with this?

Answer 1

Complexity Parameter CP describes a threshold $T$. If branch provides improvement less than $T$ it is deleted from the tree. You are using cp=0 so you told the algorithm to DO NOT prune any branches independly on their results! CP tested by printcp are very small also.

The bigger value of CP is, the more branches would be pruned so your tree should be smaller and should generalize better. Therefore you can try with highers values of CP at start and then print cross-validation results for getting suggested CP value and finally use prune method to reduce your tree for example:

model_pruned <- prune(rpart_model, cp=0.01)

You can compare your trees' structures using rpart.plot(tree_object) method. I think the original tree is way too complex and that is why it overfitts at start.