I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features.
By performing 5-fold cross-validation on different sized subsets of the data set, I have found a strong relationship between the fraction of the data set used and the value of both the R-squared metric and the RMSE metric.
EDIT for clarity: for a fraction of 0.01, I am taking 1/100 of the samples (~60) and then performing repeated 5-fold cross-validation as if this were the whole data set. The values shown are means.
It appears that RMSE varies linearly with the reciprocal of the fraction used (1/Frac), and the R-squared regresses very well against a second-order polynomial of 1/Frac.
The raw results are as follows:
Frac 1/Frac Test R2 Test RMSE 0.01 100 -1.65628 1.85292 0.013 75 -0.71208 1.71476 0.02 50 -0.00786883 1.38874 0.05 20 0.535839 1.06872 0.10 10 0.626532 1.00598 0.20 5 0.702421 0.95058 0.30 3.33 0.745277 0.860082 0.50 2 0.772211 0.86548
My questions are:
- Is this a well-known relationship?
- Does it have a name (so I can research further)?
- Is there a theoretical basis for it?