I'm fitting gradient boosted decision trees (lightgbm) to model a regression problem. The data is extremely noisy, $R^2 \approx 0$.
I'm trying to improve the fitting procedure using 10 fold cross validation.
I plot the learning curves, that is, training score and validation score as a function of number of iterations (=number of trees) for each train/validation split.
The train scores are increasing in each split.
But for half the splits, the test scores are increasing, and for the other half they're decreasing.
Does this mean that on half the splits I'm underfitting, the other half I'm overfitting? If so, what can I do?