There's a not well-known fact in statistics. That as your sample size increases, more p-values become significant.
I'm working with a massive sample size consisting of 3 million samples (~ 1% of the U.S. population). It's a Logistic Regression, and p-values of several relations are significant at 5%, 1% and ~0%.
I fielded my paper's abstract to several journals to see if there was interest. Only a German journal I submitted to (those smart Germans, really) caught on that an extremely large sample size like this might more easily produce significant p-values and that I'd need to somehow adjust for that.
To be clear, an excessively large sample size doesn't produce spurious p-values. It's that more effects with tiny effect sizes start showing significant p-values.
I want your views on strategies for how to deal with such a situation, to produce reliable research. Some strategies I've thought of are:
Conduct the research as is, and only consider as relevant results that have effect sizes that are above a certain threshold (What threshold though?)
Do what they do in Data Mining. Split up my data into train, validate, and test sets. (Is this done with Logistic Regression though? Can anyone point me to past papers where the technique is illustrated?)
Skip the usual Logistic Regression and instead use a Data Mining technique such as CART (which may also use Logistic Regression internally).
I'd like your views on the relative worth of these strategies. Other suggestions, too, would be welcome. Especially welcome will be pointers to prior papers where the authors illustrate how to deal with a similar problem.