2
$\begingroup$

I have a not-quite linear regression problem which I am investigating. The data set is fairly large, with ~6000 samples and ~2100 features.

By performing 5-fold cross-validation on different sized subsets of the data set, I have found a strong relationship between the fraction of the data set used and the value of both the R-squared metric and the RMSE metric.

EDIT for clarity: for a fraction of 0.01, I am taking 1/100 of the samples (~60) and then performing repeated 5-fold cross-validation as if this were the whole data set. The values shown are means.

It appears that RMSE varies linearly with the reciprocal of the fraction used (1/Frac), and the R-squared regresses very well against a second-order polynomial of 1/Frac.

The raw results are as follows:

Frac   1/Frac  Test R2      Test RMSE
0.01   100     -1.65628     1.85292
0.013  75      -0.71208     1.71476
0.02   50      -0.00786883  1.38874
0.05   20       0.535839    1.06872
0.10   10       0.626532    1.00598
0.20   5        0.702421    0.95058
0.30   3.33     0.745277    0.860082
0.50   2        0.772211    0.86548

Relationships:

Relationship between 1/Frac and R-squared

Relationship between 1/Frac and RMSE

My questions are:

  1. Is this a well-known relationship?
  2. Does it have a name (so I can research further)?
  3. Is there a theoretical basis for it?
$\endgroup$
3
$\begingroup$

Do I understand correctly from your test RMSE, that the error is lower as you increase the size of your fraction used in training?

If you do 5-fold cross validation using a fraction of 0.01, that implies here that you are only a total fraction of only 0.05 of your data? So 0.04 for training and 0.01 for testing.

It should not come as a surprise then, that the performance improves with increased amounts of data used for training. Your RMSE values increase with fraction size - I would normally expect such a correlation (assuming your data is fairly coherent).

EDIT:

To answer your questions,

  1. I would argue it is a well-known relationship, even in the simplest of regression problems. Have a read here (including comments on accepted answer) for more discussion on minimum required data points for valid models. One reason highlighted is that very few data points makes any estimate of variance less meaningful

  2. I don't think there is a name given to this relationship - at least I don't know of one (and neither do people in the link above). I would just describe it as being inherent to problem that the dataset is trying to represent. For example, if we have a high-dimensional complex parameter space, and then only a handful of data-points sampled from that space, it is very unlikely that we can fully describe the original space. The more samples we have, the closer we can approximate it. This is extremely evident in data-intensive modelling techniques, such as image-based tasks (i.e. convolutional neural networks, with millions of parameters.) Other methodologies, such as Bayesian optimisation using Gaussian processes. These are computationally more intense, but perhaps offer higher data efficiency.

  3. There might be some convergence theories, stating how approximations to ground truth reach a threshold given a certain amount of data (that kind of framework would make sense to me) - but alas, I have not seen any formal analysis in this direction. Regarding number of samples, in the meantime, I just follow the mantra: the more the merrier!

$\endgroup$
  • $\begingroup$ I will clarify the original post, but for a fraction of 0.01, this means that I am taking 1/100th of the original data points and then performing 5-fold cross-validation as if that was the complete data set. $\endgroup$ – Matt Wenham Jun 25 at 12:28
  • $\begingroup$ @MattWenham - In that case I guess I would stick by my point... if you use a lot less data (0.01 * 6000 = 60 sample) in your large feature-space (~2100 dimensional), then I would expect higher errors on smaller datasets. $\endgroup$ – n1k31t4 Jun 25 at 12:33
  • 1
    $\begingroup$ Granted, I expected to find a such relationship. But is the strong linear or polynomial relationship I have found well-known, and does it have a basis in theory? That is the crux of my question. $\endgroup$ – Matt Wenham Jun 25 at 12:43
  • $\begingroup$ Thanks for your clarification, very useful. What I am ultimately speculating on is whether it is valid to use the intercept points of regressions like the two I performed above to estimate the value of the metric for an infinite (or at least for some value of large) amount of data more quickly. This could speed up the cross-validation process considerably. $\endgroup$ – Matt Wenham Jun 25 at 14:14
  • $\begingroup$ This phenomenon is often refered to as a learning curve, which is just a plot with your loss on the y axis and the number of observations on the x axis. These plots often tail off after a certain amount of data is used. In general it it thought that neural networks and other high performing algorithms are often superior to simpler models (for large datasets) because they tail off much slower; that is, they can take advantage of large amounts of data compared to say a GLM that doesn't improve much after comparably lower number of observations in the training set which is also a good thing... $\endgroup$ – aranglol Jun 25 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.