# When is there enough data for generalization?

Are there any general rules that one can use to infer what can be learned/generalized from a particular data set? Suppose the dataset was taken from a sample of people. Can these rules be stated as functions of the sample or total population?

I understand the above may be vague, so a case scenario: Users participate in a search task, where the data are their queries, clicked results, and the HTML content (text only) of those results. Each of these are tagged with their user and timestamp. A user may generate a few pages - for a simple fact-finding task - or hundreds of pages - for a longer-term search task, like for class report.

Edit: In addition to generalizing about a population, given a sample, I'm interested in generalizing about an individual's overall search behavior, given a time slice. Theory and paper references are a plus!

• There's one way to get "generalization" that is good enough for any scenario - sample the entire population. In all other cases your best option is to select confidence level and take sample large enough to give reasonable confidence interval. Commented Aug 4, 2014 at 20:06

It is my understanding that random sampling is a mandatory condition for making any generalization statements. IMHO, other parameters, such as sample size, just affect probability level (confidence) of generalization. Furthermore, clarifying the @ffriend's comment, I believe that you have to calculate needed sample size, based on desired values of confidence interval, effect size, statistical power and number of predictors (this is based on Cohen's work - see References section at the following link). For multiple regression, you can use the following calculator: http://www.danielsoper.com/statcalc3/calc.aspx?id=1.

More information on how to select, calculate and interpret effect sizes can be found in the following nice and comprehensive paper, which is freely available: http://jpepsy.oxfordjournals.org/content/34/9/917.full.

If you're using R (and even, if you don't), you may find the following Web page on confidence intervals and R interesting and useful: http://osc.centerforopenscience.org/static/CIs_in_r.html.

Finally, the following comprehensive guide to survey sampling can be helpful, even if you're not using survey research designs. In my opinion, it contains a wealth of useful information on sampling methods, sampling size determination (including calculator) and much more: http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm.

There are two rules for generalizability:

1. The sample must be representative. In expectation, at least, the distribution of features in your sample must match the distribution of features in the population. When you are fitting a model with a response variable, this includes features that you do not observe, but that affect any response variables in your model. Since it is, in many cases, impossible to know what you do not observe, random sampling is used.

The idea with randomization is that a random sample, up to sampling error, must accurately reflect the distribution of all features in the population, observed and otherwise. This is why randomization is the "gold standard," but if sample control is available by some other technique, or it is defensible to argue that there are no omitted features, then it isn't always necessary.

2. Your sample must be large enough that the effect of sampling error on the feature distribution is relatively small. This is, again, to ensure representativeness. But deciding who to sample is different from deciding how many people to sample.

Since it sounds like you're fitting a model, there's the additional consideration that certain important combinations of features could be relatively rare in the population. This is not an issue for generalizability, but it bears heavily on your considerations for sample size. For instance, I'm working on a project now with (non-big) data that was originally collected to understand the experiences of minorities in college. As such, it was critically important to ensure that statistical power was high specifically in the minority subpopulation. For this reason, blacks and Latinos were deliberately oversampled. However, the proportion by which they were oversampled was also recorded. These are used to compute survey weights. These can be used to re-weight the sample so as to reflect the estimated population proportions, in the event that a representative sample is required.

An additional consideration arises if your model is hierarchical. A canonical use for a hierarchical model is one of children's behavior in schools. Children are "grouped" by school and share school-level traits. Therefore a representative sample of schools is required, and within each school a representative sample of children is required. This leads to stratified sampling. This and some other sampling designs are reviewed in surprising depth on Wikipedia.

To answer a simpler, but related question, namely 'How well can my model generalize on the data that I have?' the method of learning curves might be applicable. This is a lecture given by Andrew Ng about them.

The basic idea is to plot test set error and training set error vs. the complexity of the model you are using (this can be somewhat complicated). If the model is powerful enough to fully 'understand' your data, at some point the complexity of the model will be high enough that performance on the training set will be close to perfect. However, the variance of a complex model will likely cause the test set performance to increase at some point.

This analysis tells you two main things, I think. The first is an upper limit on performance. It's pretty unlikely that you'll do better on data that you haven't seen than on your training data. The other thing it tells you is whether or not getting more data might help. If you can demonstrate that you fully understand your training data by driving training error to zero it might be possible, through the inclusion of more data, to drive your test error further down by getting a more complete sample and then training a powerful model on that.