Strategy for dealing with giant sample size

Question

There's a not well-known fact in statistics. That as your sample size increases, more p-values become significant.

I'm working with a massive sample size consisting of 3 million samples (~ 1% of the U.S. population). It's a Logistic Regression, and p-values of several relations are significant at 5%, 1% and ~0%.

I fielded my paper's abstract to several journals to see if there was interest. Only a German journal I submitted to (those smart Germans, really) caught on that an extremely large sample size like this might more easily produce significant p-values and that I'd need to somehow adjust for that.

To be clear, an excessively large sample size doesn't produce spurious p-values. It's that more effects with tiny effect sizes start showing significant p-values.

I want your views on strategies for how to deal with such a situation, to produce reliable research. Some strategies I've thought of are:

1. Conduct the research as is, and only consider as relevant results that have effect sizes that are above a certain threshold (What threshold though?)

2. Do what they do in Data Mining. Split up my data into train, validate, and test sets. (Is this done with Logistic Regression though? Can anyone point me to past papers where the technique is illustrated?)

3. Skip the usual Logistic Regression and instead use a Data Mining technique such as CART (which may also use Logistic Regression internally).

I'd like your views on the relative worth of these strategies. Other suggestions, too, would be welcome. Especially welcome will be pointers to prior papers where the authors illustrate how to deal with a similar problem.

Answer 1

My link in the comment has useful advice. I'd like to emphasis:

• This is a very well known fact in statistics
• Large sample size is good. There's nothing wrong with more and better quality data.
• It's silly to split up the data in your second point. You are voluntarily giving away information and statistical power. People do that for training an unbiased model, but not to reduce sample size for the p-values.
• There's no reason to switch to another model. The standard error in your logistic model should be small, which makes it ideal for predictive modelling. Your model should be robust and stable.
• You should do your analysis on effects size if you have a large sample. Read @Sympa's answer in the link I post. What's your magnitude of the effect size? What's the relative effects?
• The threshold you have for your first point is domain specific. If you write a paper, you should know it very well. You could compare your data with the literature. Simple descriptive statistics like percentage and how it compare with the standard deviation (which is independent to sample size) would be useful.

Please also take a look at https://stats.stackexchange.com/questions/125750/sample-size-too-large.

There is a long but excellent article on Bayes factor and p-value:

https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-with-bayes-factors-a-miracle-cure-for-the-replicability-crisis-in-psychological-science/

I should draw to your attention the following paragraph:

The more interesting argument against p-value is not that significant results in large studies are type-I errors, but that these results are practically meaningless. To make this point, statistics books often distinguish statistical significance and practical significance and warn that statistically significant results in large samples may have little practical significance. This warning was useful in the past when researchers would only report p-values (e.g., women have higher verbal intelligence than men, p < .05). The p-value says nothing about the size of the effect. When only the p-value is available, it makes sense to assume that significant results in smaller samples are larger because only large effects can be significant in these samples. However, large effects can also be significant in large samples and large effects in small studies can be inflated by sampling error. Thus, the notion of practical significance is outdated and should be replaced by questions about effect sizes. Neither p-values nor Bayes-Factors provide information about the size of the effect or the practical implications of a finding.

The reviewer was correct. Neither p-value nor the bayes-factor reveals anything about the effect sizes.

Answer 2

How easy is it to formulate a null-hypothesis / random data? If it is possible you can just see how often a certain p-value is calculated. By creating enough pseudo-experiments you can arrive at a probability density function of p-values which you can use to compare your measured ones. You can do this for one variable alone but also for several variables combined. Example: How likely is it to find 10 variables with a p-value < x when I have 100 variables.

Answer 3

Hypothesis tests of large amounts of data can identify tiny effects, yes. I disagree, however, that the one journal showed particularly great insight. In fact, I would argue that the journal botched the statistics, since this is a feature, not a bug, of hypothesis testing.

You don’t have to adjust for anything. The test is doing exactly what it should be doing and what we want it to do.

If you are concerned about the test flagging a difference as statistically significant when it is not practically important, then your knowledge of the subject matter (medicine, geology, whatever) can guide you to what is important enough for your field to find interesting. Your analysis is allowed to conclude something like, “…while this result shows strong statistical significance, the magnitude is too small for us to care” (use better phrasing, but that is the gist).